Oct 30 01:28:18.736367 ip-10-0-130-0 systemd[1]: Starting Kubernetes Kubelet... Oct 30 01:28:19.132134 ip-10-0-130-0 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:28:19.132134 ip-10-0-130-0 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 01:28:19.132134 ip-10-0-130-0 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:28:19.132134 ip-10-0-130-0 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:28:19.137426 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137343 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:19.137426 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137425 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:19.137426 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137429 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137433 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137435 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137438 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137441 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137444 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137446 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137450 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137453 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137456 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137458 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137461 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137464 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137467 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137469 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137478 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137481 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137484 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137486 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:19.137521 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137489 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137492 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137495 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137497 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137500 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137503 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137506 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137509 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137511 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137517 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137521 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137524 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137526 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137529 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137533 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137535 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137538 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137540 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137549 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137551 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:19.137990 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137554 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137557 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137559 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137562 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137564 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137567 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137570 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137572 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137575 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137578 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137580 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137583 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137586 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137589 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137592 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137595 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137597 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137600 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137602 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137605 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:19.138476 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137608 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137611 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137614 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137616 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137619 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137621 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137624 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137627 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137629 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137633 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137638 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137647 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137650 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137652 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137657 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137661 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137665 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137667 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137670 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:19.138936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137673 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137675 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137678 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137681 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137684 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137686 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137689 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137692 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137695 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137697 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137700 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137702 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137706 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137709 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137711 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137714 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137717 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137719 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137722 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:19.139393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137724 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137727 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137730 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137732 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137736 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137739 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.137742 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138586 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138595 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138598 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138601 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138604 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138607 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138609 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138612 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138614 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138617 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138621 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138624 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138627 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:19.139836 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138631 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138635 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138638 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138641 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138644 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138647 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138650 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138652 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138655 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138658 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138660 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138663 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138665 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138668 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138671 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138673 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138676 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138679 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138681 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:19.140355 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138685 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138687 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138690 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138692 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138695 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138698 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138700 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138703 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138706 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138708 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138712 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138714 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138717 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138719 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138722 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138725 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138727 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138730 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138733 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138736 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:19.140817 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138738 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138743 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138745 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138748 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138750 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138753 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138756 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138758 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138761 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138764 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138766 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138769 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138772 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138774 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138777 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138780 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138782 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138785 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138787 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138790 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:19.141323 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138792 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138795 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138798 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138801 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138803 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138806 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138808 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138811 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138813 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138816 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138819 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138822 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138825 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138827 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138829 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138832 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138834 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138837 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138839 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138842 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:19.141798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138844 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138847 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138849 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138852 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138854 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138858 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138863 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138866 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138868 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138871 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138873 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138876 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138879 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.138881 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139411 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139418 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139422 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139425 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139428 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:19.142271 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139430 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139433 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139436 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139440 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139443 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139446 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139449 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139452 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139455 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139458 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139460 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139463 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139466 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139468 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139471 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139473 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139476 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139478 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139481 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139484 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:19.142798 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139487 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139490 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139493 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139495 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139498 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139501 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139503 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139506 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139509 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139512 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139515 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139517 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139520 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139522 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139525 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139528 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139530 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139533 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139535 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139538 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:19.143282 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139540 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139543 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139545 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139548 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139550 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139553 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139556 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139558 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139560 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139563 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139565 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139568 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139571 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139573 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139576 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139579 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139581 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139584 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139586 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:19.143750 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139590 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139594 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139597 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139602 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139605 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139608 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139611 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139613 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139619 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139622 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139624 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139627 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139629 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139632 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139635 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139637 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139639 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139642 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139645 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139647 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:19.144204 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139650 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139652 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139655 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139657 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139660 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139663 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139666 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139669 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139671 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139674 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139677 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139679 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139682 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139684 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139687 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139690 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139693 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139695 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139698 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139700 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:19.144680 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139704 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.139707 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139795 2582 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139802 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139811 2582 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139818 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139826 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139830 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139837 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139842 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139846 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139849 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139853 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139856 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139859 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139863 2582 flags.go:64] FLAG: --cgroup-root="" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139866 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139869 2582 flags.go:64] FLAG: --client-ca-file="" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139872 2582 flags.go:64] FLAG: --cloud-provider="external" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139876 2582 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139880 2582 flags.go:64] FLAG: --cluster-domain="" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139884 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139887 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 01:28:19.145170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139890 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139894 2582 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139898 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139901 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139904 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139907 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139910 2582 flags.go:64] FLAG: --contention-profiling="false" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139913 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139917 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139922 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139925 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139929 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139932 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139935 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139939 2582 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139943 2582 flags.go:64] FLAG: --enable-server="true" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139947 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139954 2582 flags.go:64] FLAG: --event-burst="100" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139958 2582 flags.go:64] FLAG: --event-qps="50" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139961 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139964 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139967 2582 flags.go:64] FLAG: --eviction-hard="" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139971 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139974 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139978 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 01:28:19.145738 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139981 2582 flags.go:64] FLAG: --eviction-soft="" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139984 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139987 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139990 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139993 2582 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139996 2582 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.139999 2582 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140002 2582 flags.go:64] FLAG: --feature-gates="" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140006 2582 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140010 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140013 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140016 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140019 2582 flags.go:64] FLAG: --healthz-port="10248" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140022 2582 flags.go:64] FLAG: --help="false" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140025 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140029 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140033 2582 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140036 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140039 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140043 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140046 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140049 2582 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140053 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 01:28:19.146392 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140056 2582 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140059 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140063 2582 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140065 2582 flags.go:64] FLAG: --kube-reserved="" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140068 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140071 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140074 2582 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140077 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140080 2582 flags.go:64] FLAG: --lock-file="" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140083 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140097 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140101 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140106 2582 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140109 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140113 2582 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140116 2582 flags.go:64] FLAG: --logging-format="text" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140119 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140122 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140125 2582 flags.go:64] FLAG: --manifest-url="" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140128 2582 flags.go:64] FLAG: --manifest-url-header="" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140133 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140135 2582 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140139 2582 flags.go:64] FLAG: --max-pods="110" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140142 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140145 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 01:28:19.146921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140150 2582 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140153 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140157 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140160 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140162 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140169 2582 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140174 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140177 2582 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140180 2582 flags.go:64] FLAG: --pod-cidr="" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140183 2582 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140186 2582 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140189 2582 flags.go:64] FLAG: --pods-per-core="0" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140192 2582 flags.go:64] FLAG: --port="10250" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140195 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140198 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04d13b4367f94ae39" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140201 2582 flags.go:64] FLAG: --qos-reserved="" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140204 2582 flags.go:64] FLAG: --read-only-port="10255" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140207 2582 flags.go:64] FLAG: --register-node="true" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140210 2582 flags.go:64] FLAG: --register-with-taints="" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140214 2582 flags.go:64] FLAG: --registry-burst="10" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140217 2582 flags.go:64] FLAG: --registry-qps="5" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140220 2582 flags.go:64] FLAG: --reserved-cpus="" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140223 2582 flags.go:64] FLAG: --reserved-memory="" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140227 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140230 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140233 2582 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 01:28:19.147517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140236 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140239 2582 flags.go:64] FLAG: --runonce="false" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140242 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140245 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140248 2582 flags.go:64] FLAG: --seccomp-default="false" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140251 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140255 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140258 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140262 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140265 2582 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140268 2582 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140271 2582 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140275 2582 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140278 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140282 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140285 2582 flags.go:64] FLAG: --system-cgroups="" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140288 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140293 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140296 2582 flags.go:64] FLAG: --tls-cert-file="" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140299 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140303 2582 flags.go:64] FLAG: --tls-min-version="" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140305 2582 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140308 2582 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140311 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140314 2582 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 01:28:19.148140 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140317 2582 flags.go:64] FLAG: --v="2" Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140321 2582 flags.go:64] FLAG: --version="false" Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140326 2582 flags.go:64] FLAG: --vmodule="" Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140330 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140333 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140464 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140468 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140471 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140474 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140476 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140481 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140484 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140488 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140492 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140495 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140498 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140501 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140504 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140506 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140511 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:19.148722 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140514 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140517 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140520 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140522 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140525 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140528 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140530 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140533 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140536 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140538 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140541 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140544 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140546 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140549 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140553 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140556 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140559 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140563 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140567 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:19.149394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140570 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140573 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140575 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140578 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140581 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140583 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140588 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140591 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140594 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140596 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140599 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140602 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140606 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140608 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140611 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140615 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140617 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140620 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140623 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140625 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:19.150102 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140628 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140631 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140633 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140637 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140639 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140643 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140645 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140648 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140650 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140653 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140656 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140659 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140661 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140664 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140667 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140669 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140672 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140675 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140678 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:19.150593 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140681 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140684 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140686 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140689 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140692 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140695 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140698 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140701 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140703 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140706 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140709 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140712 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140715 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140717 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140720 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140722 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140733 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140736 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140738 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140741 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:19.151071 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140743 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:19.151560 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140746 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:19.151560 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140749 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:19.151560 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140752 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:19.151560 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140755 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:19.151560 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140758 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:19.151560 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140761 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:19.151560 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140764 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:19.151560 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140766 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:19.151560 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140769 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:19.151560 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140771 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:19.151560 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140776 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:19.151560 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140779 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:19.151560 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.140805 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:28:19.151560 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140846 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:19.151560 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140851 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:19.151560 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140854 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140857 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140861 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140864 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140866 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140869 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140872 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140874 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140877 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140880 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140884 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140888 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140891 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140894 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140897 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140900 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140902 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140905 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140907 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:19.151941 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140910 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140912 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140915 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140918 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140921 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140923 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140926 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140929 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140933 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140936 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140938 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140941 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140944 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140946 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140949 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140953 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140955 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140958 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140960 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140963 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:19.152394 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140966 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140968 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140971 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140974 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140976 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140979 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140981 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140984 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140986 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140989 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140991 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140994 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140996 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.140999 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141001 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141004 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141007 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141010 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141012 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:19.152871 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141015 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141019 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141021 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141024 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141027 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141029 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141032 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141035 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141040 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141042 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141045 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141048 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141050 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141053 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141055 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141058 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141060 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141063 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141066 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141068 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:19.153327 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141071 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141073 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141076 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141078 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141081 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141083 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141100 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141103 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141105 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141108 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141112 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141114 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141117 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141121 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141126 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141130 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141133 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141135 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141138 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:19.153792 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141141 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141147 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141149 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141152 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141155 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141157 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.141160 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.147579 2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.147594 2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.147608 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.147613 2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147670 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147675 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147679 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147682 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147685 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147688 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147691 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147694 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147696 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:19.154252 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147699 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147702 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147705 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147709 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147711 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147715 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147718 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147720 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147723 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147726 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147728 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147731 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147734 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147736 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147740 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147742 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147745 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147748 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147750 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147752 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:19.154760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147757 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147761 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147765 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147768 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147771 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147774 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147777 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147780 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147782 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147785 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147788 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147790 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147793 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147796 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147798 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147801 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147804 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147806 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147809 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:19.155304 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147812 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147814 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147817 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147819 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147822 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147824 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147826 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147829 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147832 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147834 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147837 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147839 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147841 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147846 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147849 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147851 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147854 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147856 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147859 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147861 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:19.155735 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147864 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147866 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147869 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147872 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147874 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147877 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147879 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147882 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147884 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147888 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147893 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147896 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147899 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147901 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147904 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147907 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147909 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147911 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147914 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147917 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:19.156208 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147919 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:19.156665 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147922 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:19.156665 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147924 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:19.156665 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147927 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:19.156665 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147929 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:19.156665 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147933 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:19.156665 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147935 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:19.156665 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147938 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:19.156665 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147940 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:19.156665 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147943 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:19.156665 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147946 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:19.156665 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147948 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:19.156665 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147951 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:19.156665 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147953 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:19.156665 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147956 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:19.156665 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147959 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:19.156665 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147961 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:19.156665 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.147964 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:19.157072 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.147992 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:28:19.157072 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148144 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:19.157072 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148149 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:19.157072 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148152 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:19.157072 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148154 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:19.157072 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148158 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:19.157072 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148161 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:19.157072 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148163 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:19.157072 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148166 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:19.157072 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148168 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:19.157072 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148171 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:19.157072 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148174 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:19.157072 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148176 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:19.157072 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148179 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:19.157072 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148181 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148183 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148186 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148189 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148192 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148194 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148197 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148199 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148202 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148205 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148207 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148210 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148212 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148215 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148217 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148220 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148222 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148225 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148229 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:19.157436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148232 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148235 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148237 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148240 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148243 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148246 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148249 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148251 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148253 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148256 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148259 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148261 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148264 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148266 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148269 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148271 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148275 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148279 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148282 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:19.157936 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148285 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148288 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148291 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148293 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148296 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148298 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148301 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148303 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148306 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148309 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148311 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148314 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148316 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148319 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148322 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148324 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148327 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148329 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148332 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148335 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:19.158393 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148337 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148340 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148342 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148345 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148347 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148349 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148352 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148355 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148357 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148360 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148362 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148365 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148368 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148370 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148373 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148375 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148378 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148381 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148383 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148386 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:19.158873 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148389 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:19.159344 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148391 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:19.159344 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148394 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:19.159344 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148396 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:19.159344 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148399 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:19.159344 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148401 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:19.159344 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148404 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:19.159344 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148406 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:19.159344 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148409 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:19.159344 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148411 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:19.159344 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148414 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:19.159344 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148417 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:19.159344 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148420 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:19.159344 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148422 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:19.159344 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148425 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:19.159344 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.148447 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:28:19.159344 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148482 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148487 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148490 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148493 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148496 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148499 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148502 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148505 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148508 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148510 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148513 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148515 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148518 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148520 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148523 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148526 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148528 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148531 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148534 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:28:19.159760 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148536 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148539 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148541 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148544 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148546 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148549 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148551 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148554 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148556 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148559 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148562 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148565 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148568 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148571 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148573 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148577 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148581 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148585 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148588 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:28:19.160266 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148591 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148594 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148597 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148599 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148602 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148604 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148607 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148609 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148612 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148615 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148617 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148620 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148622 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148625 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148628 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148633 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148636 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148641 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148644 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:28:19.160694 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148647 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148650 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148652 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148656 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148659 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148674 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148678 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148680 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148683 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148686 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148689 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148691 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148694 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148697 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148701 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148704 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148706 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148709 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148711 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:28:19.161199 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148714 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148717 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148719 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148722 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148733 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148736 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148739 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148742 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148745 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148747 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148750 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148754 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148756 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148759 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148762 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148764 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148767 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148769 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148772 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148775 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:28:19.161645 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148778 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:28:19.162126 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148781 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:28:19.162126 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148784 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:28:19.162126 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148787 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:28:19.162126 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148789 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:28:19.162126 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148792 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:28:19.162126 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148794 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:28:19.162126 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148798 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:28:19.162126 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148801 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:28:19.162126 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:19.148804 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:28:19.162126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.148855 2582 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 01:28:19.162126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.152453 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 01:28:19.162126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.153364 2582 server.go:1014] "Starting client certificate rotation" Oct 30 01:28:19.162126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.153466 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 01:28:19.162126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.153506 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 01:28:19.162126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.155595 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:28:19.162126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.159383 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:28:19.170122 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.170106 2582 log.go:25] "Validated CRI v1 runtime API" Oct 30 01:28:19.173123 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.173108 2582 log.go:25] "Validated CRI v1 image API" Oct 30 01:28:19.173908 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.173885 2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 01:28:19.176259 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.176242 2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 95bd4926-4597-4527-8411-f8edc5a3c90e:/dev/nvme0n1p3 ee1a38bd-3918-482f-acad-028e5cfeebb0:/dev/nvme0n1p4] Oct 30 01:28:19.176305 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.176259 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 01:28:19.180476 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.180341 2582 manager.go:217] Machine: {Timestamp:2026-10-30 01:28:19.179853863 +0000 UTC m=+0.316729950 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3118290 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20bd49eca7107ded8fd169f2678842 SystemUUID:ec20bd49-eca7-107d-ed8f-d169f2678842 BootID:c09f8dcd-a2a9-416b-98f0-f03c48b6945f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f6:42:39:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f6:42:39:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:65:8b:a0:0e:0b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 01:28:19.180476 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.180475 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 01:28:19.180569 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.180548 2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 01:28:19.183521 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.183487 2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 01:28:19.183656 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.183523 2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-0.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 01:28:19.183697 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.183658 2582 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 01:28:19.183697 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.183666 2582 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 01:28:19.183697 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.183679 2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:28:19.183697 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.183692 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:28:19.184488 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.184478 2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 01:28:19.188067 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.188055 2582 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 01:28:19.188198 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.188188 2582 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 01:28:19.188898 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.188889 2582 kubelet.go:493] "Attempting to sync node with API server" Oct 30 01:28:19.188929 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.188903 2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 01:28:19.188929 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.188913 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 01:28:19.188929 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.188922 2582 kubelet.go:405] "Adding apiserver pod source" Oct 30 01:28:19.189026 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.188930 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 01:28:19.189266 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.189252 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:28:19.192711 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.192687 2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 01:28:19.199941 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.199926 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:28:19.200673 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.200659 2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 01:28:19.200769 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.200689 2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 01:28:19.201253 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.201241 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 01:28:19.201317 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.201257 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 01:28:19.201317 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.201264 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 01:28:19.201317 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.201270 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 01:28:19.201317 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.201275 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 01:28:19.201317 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.201281 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 01:28:19.201863 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.201855 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 01:28:19.201898 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.201863 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 01:28:19.201898 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.201872 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 01:28:19.201898 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.201879 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 01:28:19.201898 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.201888 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 01:28:19.201898 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.201897 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 01:28:19.202463 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.202454 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 01:28:19.202463 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.202463 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 01:28:19.206192 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.206179 2582 server.go:1263] "Started kubelet" Oct 30 01:28:19.206314 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.206281 2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 01:28:19.206350 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.206288 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 01:28:19.206378 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.206360 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 01:28:19.206951 ip-10-0-130-0 systemd[1]: Started Kubernetes Kubelet. Oct 30 01:28:19.207712 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.207695 2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 01:28:19.209986 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.209968 2582 server.go:317] "Adding debug handlers to kubelet server" Oct 30 01:28:19.215107 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.215073 2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 01:28:19.215209 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.215083 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 01:28:19.215313 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.215147 2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 01:28:19.215313 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.215304 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-0.ec2.internal\" not found" Oct 30 01:28:19.215901 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.215885 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 01:28:19.216019 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.215886 2582 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 01:28:19.216019 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.216021 2582 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 01:28:19.216197 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.215979 2582 factory.go:55] Registering systemd factory Oct 30 01:28:19.216197 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.216118 2582 factory.go:223] Registration of the systemd container factory successfully Oct 30 01:28:19.216197 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.216184 2582 reconstruct.go:97] "Volume reconstruction finished" Oct 30 01:28:19.216197 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.216192 2582 reconciler.go:29] "Reconciler: start to sync state" Oct 30 01:28:19.216384 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.216334 2582 factory.go:153] Registering CRI-O factory Oct 30 01:28:19.216384 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.216343 2582 factory.go:223] Registration of the crio container factory successfully Oct 30 01:28:19.216450 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.216392 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 01:28:19.216450 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.216411 2582 factory.go:103] Registering Raw factory Oct 30 01:28:19.216450 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.216424 2582 manager.go:1199] Started watching for new ooms in manager Oct 30 01:28:19.217304 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.217287 2582 manager.go:319] Starting recovery of all containers Oct 30 01:28:19.227999 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.227977 2582 manager.go:324] Recovery completed Oct 30 01:28:19.233950 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.233351 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:19.235373 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.235350 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:19.235436 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.235381 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:19.235436 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.235391 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:19.235940 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.235920 2582 cpu_manager.go:226] "Starting" policy="none" Oct 30 01:28:19.235940 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.235931 2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 01:28:19.235940 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.235922 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:19.236113 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.235947 2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 01:28:19.236785 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.236761 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-0.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 01:28:19.236785 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.236760 2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-0.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 01:28:19.236904 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.236857 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:19.236904 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.236883 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:19.236904 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.236897 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:19.237821 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.236833 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-0.ec2.internal.18e32a672a78ea52 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-0.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-0.ec2.internal,},FirstTimestamp:2026-10-30 01:28:19.206154834 +0000 UTC m=+0.343030920,LastTimestamp:2026-10-30 01:28:19.206154834 +0000 UTC m=+0.343030920,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-0.ec2.internal,}" Oct 30 01:28:19.238235 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.238218 2582 policy_none.go:50] "Start" Oct 30 01:28:19.238310 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.238240 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 01:28:19.238310 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.238261 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 01:28:19.251622 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.251597 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 01:28:19.251938 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.251921 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-0.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 01:28:19.251938 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.251931 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 01:28:19.254256 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.254236 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 01:28:19.254334 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.254274 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t5p9n" Oct 30 01:28:19.263382 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.263297 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-0.ec2.internal.18e32a672c36a30e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-0.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-0.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-0.ec2.internal,},FirstTimestamp:2026-10-30 01:28:19.235365646 +0000 UTC m=+0.372241734,LastTimestamp:2026-10-30 01:28:19.235365646 +0000 UTC m=+0.372241734,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-0.ec2.internal,}" Oct 30 01:28:19.270034 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.270014 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t5p9n" Oct 30 01:28:19.278431 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.278356 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-0.ec2.internal.18e32a672c36f1d9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-0.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-0.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-0.ec2.internal,},FirstTimestamp:2026-10-30 01:28:19.235385817 +0000 UTC m=+0.372261904,LastTimestamp:2026-10-30 01:28:19.235385817 +0000 UTC m=+0.372261904,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-0.ec2.internal,}" Oct 30 01:28:19.315632 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.315607 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-0.ec2.internal\" not found" Oct 30 01:28:19.377222 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.377199 2582 policy_none.go:44] "Start" Oct 30 01:28:19.405710 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.405676 2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 01:28:19.408016 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.407999 2582 manager.go:341] "Starting Device Plugin manager" Oct 30 01:28:19.408552 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.408536 2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 01:28:19.408552 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.408553 2582 server.go:87] "Starting device plugin registration server" Oct 30 01:28:19.408791 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.408781 2582 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 01:28:19.408842 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.408792 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 01:28:19.408933 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.408914 2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 01:28:19.409050 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.409033 2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 01:28:19.409050 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.409044 2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 01:28:19.409597 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.409577 2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 01:28:19.409665 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.409632 2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-0.ec2.internal\" not found" Oct 30 01:28:19.465323 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.465156 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 01:28:19.467245 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.466034 2582 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-0.ec2.internal\" not found" node="ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.467245 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.466548 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 01:28:19.467245 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.466573 2582 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 01:28:19.467245 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.466597 2582 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 01:28:19.467245 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.466638 2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 01:28:19.481355 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.481333 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:19.509427 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.509391 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:19.510372 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.510354 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:19.510455 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.510392 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:19.510455 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.510408 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:19.510455 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.510435 2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.530314 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.530296 2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.530370 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.530319 2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-0.ec2.internal\": node \"ip-10-0-130-0.ec2.internal\" not found" Oct 30 01:28:19.567724 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.567695 2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal"] Oct 30 01:28:19.567840 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.567774 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:19.568631 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.568614 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:19.568703 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.568647 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:19.568703 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.568662 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:19.570825 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.570813 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:19.570986 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.570974 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.571019 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.571003 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:19.571805 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.571789 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:19.571805 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.571802 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:19.571937 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.571815 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:19.571937 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.571818 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:19.571937 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.571828 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:19.571937 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.571829 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:19.573873 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.573858 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.573942 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.573884 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:19.574765 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.574746 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:19.574853 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.574776 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:19.574853 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.574790 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:19.575125 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.575110 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-0.ec2.internal\" not found" Oct 30 01:28:19.597458 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.597442 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-0.ec2.internal\" not found" node="ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.600569 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.600552 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-0.ec2.internal\" not found" node="ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.617937 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.617917 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8da95128540617dd483ff23c5d0ddd95-config\") pod \"kube-apiserver-proxy-ip-10-0-130-0.ec2.internal\" (UID: \"8da95128540617dd483ff23c5d0ddd95\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.618055 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.617940 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f54d525f1aa0b0582cbd13c31e87b9a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal\" (UID: \"f54d525f1aa0b0582cbd13c31e87b9a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.618055 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.617959 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f54d525f1aa0b0582cbd13c31e87b9a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal\" (UID: \"f54d525f1aa0b0582cbd13c31e87b9a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.675441 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.675421 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-0.ec2.internal\" not found" Oct 30 01:28:19.718843 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.718813 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8da95128540617dd483ff23c5d0ddd95-config\") pod \"kube-apiserver-proxy-ip-10-0-130-0.ec2.internal\" (UID: \"8da95128540617dd483ff23c5d0ddd95\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.718843 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.718835 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8da95128540617dd483ff23c5d0ddd95-config\") pod \"kube-apiserver-proxy-ip-10-0-130-0.ec2.internal\" (UID: \"8da95128540617dd483ff23c5d0ddd95\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.719008 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.718861 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f54d525f1aa0b0582cbd13c31e87b9a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal\" (UID: \"f54d525f1aa0b0582cbd13c31e87b9a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.719008 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.718903 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f54d525f1aa0b0582cbd13c31e87b9a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal\" (UID: \"f54d525f1aa0b0582cbd13c31e87b9a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.719008 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.718886 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f54d525f1aa0b0582cbd13c31e87b9a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal\" (UID: \"f54d525f1aa0b0582cbd13c31e87b9a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.719008 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.718940 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f54d525f1aa0b0582cbd13c31e87b9a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal\" (UID: \"f54d525f1aa0b0582cbd13c31e87b9a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.776397 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.776370 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-0.ec2.internal\" not found" Oct 30 01:28:19.876980 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.876951 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-0.ec2.internal\" not found" Oct 30 01:28:19.900532 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.900506 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.903656 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:19.903638 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal" Oct 30 01:28:19.977700 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:19.977622 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-0.ec2.internal\" not found" Oct 30 01:28:20.078176 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:20.078152 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-0.ec2.internal\" not found" Oct 30 01:28:20.153113 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:20.153073 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 01:28:20.153570 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:20.153223 2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 01:28:20.178651 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:20.178620 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-0.ec2.internal\" not found" Oct 30 01:28:20.275198 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:20.275123 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 01:23:19 +0000 UTC" deadline="2028-07-20 17:45:35.80218269 +0000 UTC" Oct 30 01:28:20.275198 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:20.275160 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15112h17m15.527025873s" Oct 30 01:28:20.279260 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:20.279231 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-0.ec2.internal\" not found" Oct 30 01:28:20.291308 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:20.291287 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:28:20.292610 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:20.292594 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:28:20.292684 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:20.292625 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:28:20.292684 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:20.292634 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:28:20.380266 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:20.380234 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-0.ec2.internal\" not found" Oct 30 01:28:20.395704 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:20.395671 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54d525f1aa0b0582cbd13c31e87b9a6.slice/crio-c4320bb0b198f2a9176fd4a26532dc30e402064783fc6444fd2f4efc57ab46bc WatchSource:0}: Error finding container c4320bb0b198f2a9176fd4a26532dc30e402064783fc6444fd2f4efc57ab46bc: Status 404 returned error can't find the container with id c4320bb0b198f2a9176fd4a26532dc30e402064783fc6444fd2f4efc57ab46bc Oct 30 01:28:20.396376 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:20.396351 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da95128540617dd483ff23c5d0ddd95.slice/crio-1d7fdd368334d11a5e39d01f71bdac4b4c5de1013b365826129fe7082e083759 WatchSource:0}: Error finding container 1d7fdd368334d11a5e39d01f71bdac4b4c5de1013b365826129fe7082e083759: Status 404 returned error can't find the container with id 1d7fdd368334d11a5e39d01f71bdac4b4c5de1013b365826129fe7082e083759 Oct 30 01:28:20.402896 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:20.401681 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:28:20.470105 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:20.470042 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal" event={"ID":"8da95128540617dd483ff23c5d0ddd95","Type":"ContainerStarted","Data":"1d7fdd368334d11a5e39d01f71bdac4b4c5de1013b365826129fe7082e083759"} Oct 30 01:28:20.471053 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:20.471030 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" event={"ID":"f54d525f1aa0b0582cbd13c31e87b9a6","Type":"ContainerStarted","Data":"c4320bb0b198f2a9176fd4a26532dc30e402064783fc6444fd2f4efc57ab46bc"} Oct 30 01:28:20.481312 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:20.481296 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-0.ec2.internal\" not found" Oct 30 01:28:20.582043 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:20.581967 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-0.ec2.internal\" not found" Oct 30 01:28:20.585210 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:20.585195 2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:20.616348 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:20.616327 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:28:20.640629 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:20.640609 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:28:20.642912 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:20.642899 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal" Oct 30 01:28:20.661736 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:20.661714 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:28:20.739595 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:20.739569 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:20.875928 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:20.875848 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:21.144856 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.144784 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:28:21.190078 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.190048 2582 apiserver.go:52] "Watching apiserver" Oct 30 01:28:21.216619 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.216593 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 01:28:21.218925 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.218304 2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 01:28:21.218925 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.218620 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-sgm2w","openshift-network-diagnostics/network-check-target-pct74","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s","openshift-multus/network-metrics-daemon-plsqk","openshift-network-operator/iptables-alerter-p9b7v","openshift-ovn-kubernetes/ovnkube-node-4qqkx","openshift-cluster-node-tuning-operator/tuned-kjlnq","openshift-dns/node-resolver-8wd8v","openshift-image-registry/node-ca-lq7lj","kube-system/global-pull-secret-syncer-55kfb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal","kube-system/konnectivity-agent-44ftt","kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal","openshift-multus/multus-additional-cni-plugins-zrv6d"] Oct 30 01:28:21.221081 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.221060 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.223341 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.223319 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8wd8v" Oct 30 01:28:21.225575 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.225556 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.227847 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.227831 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:21.227935 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:21.227885 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:21.231491 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.231472 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p9b7v" Oct 30 01:28:21.234166 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.234149 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.236374 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.236353 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lq7lj" Oct 30 01:28:21.238602 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.238585 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.241186 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.241167 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:21.241264 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:21.241235 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:21.243705 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.243684 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:21.243813 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:21.243764 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:21.246538 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.246519 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.249243 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.248954 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-44ftt" Oct 30 01:28:21.255970 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.255313 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lm6wv\"" Oct 30 01:28:21.255970 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.255392 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j7lll\"" Oct 30 01:28:21.255970 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.255426 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rwcx9\"" Oct 30 01:28:21.255970 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.255477 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c6clm\"" Oct 30 01:28:21.255970 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.255508 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 01:28:21.255970 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.255642 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 01:28:21.255970 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.255775 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7p9xn\"" Oct 30 01:28:21.255970 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.255846 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lzbd2\"" Oct 30 01:28:21.256930 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.256905 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 01:28:21.257069 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.256932 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 01:28:21.257209 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.257079 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:21.257591 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.257572 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 01:28:21.257775 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.257752 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:28:21.257864 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.257757 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 01:28:21.257864 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.257837 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 01:28:21.257972 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.257905 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 01:28:21.257972 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.257934 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 01:28:21.259753 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.259715 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 01:28:21.260449 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.260258 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 01:28:21.260449 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.260321 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 01:28:21.267604 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.267587 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 01:28:21.267604 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.267596 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 01:28:21.267724 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.267620 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 01:28:21.267724 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.267639 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 01:28:21.267724 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.267642 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 01:28:21.267724 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.267664 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 01:28:21.268405 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.268389 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 01:28:21.268469 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.268416 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lkdgr\"" Oct 30 01:28:21.268469 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.268457 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 01:28:21.268552 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.268458 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 01:28:21.293012 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.292985 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 01:28:21.293141 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.293055 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2946c" Oct 30 01:28:21.308266 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.308243 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2946c" Oct 30 01:28:21.317361 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.317340 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 01:28:21.327213 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327186 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-cni-netd\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.327334 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327217 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0d9a754-4e82-4dd5-b423-50624589bdda-konnectivity-ca\") pod \"konnectivity-agent-44ftt\" (UID: \"a0d9a754-4e82-4dd5-b423-50624589bdda\") " pod="kube-system/konnectivity-agent-44ftt" Oct 30 01:28:21.327334 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327238 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-tmp\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.327334 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327263 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwzmn\" (UniqueName: \"kubernetes.io/projected/40d2d25c-e7a0-4ded-9674-19c6d371522e-kube-api-access-qwzmn\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.327334 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327289 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-etc-openvswitch\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.327334 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327307 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c01f6b4a-5666-4734-b403-3818692c3a74-ovnkube-config\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.327334 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327324 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-dbus\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:21.327659 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327347 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-device-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.327659 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327371 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-sys-fs\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.327659 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327431 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvtjv\" (UniqueName: \"kubernetes.io/projected/4f486e9e-aa32-4a9a-877b-eca8dbba7091-kube-api-access-wvtjv\") pod \"node-ca-lq7lj\" (UID: \"4f486e9e-aa32-4a9a-877b-eca8dbba7091\") " pod="openshift-image-registry/node-ca-lq7lj" Oct 30 01:28:21.327659 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327457 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-etc-kubernetes\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.327659 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327482 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-run\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.327659 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327502 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-conf-dir\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.327659 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327516 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-slash\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.327659 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327537 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-run-netns\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.327659 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327553 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-sysctl-d\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.327659 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327574 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-tuned\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.327659 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327604 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjjjr\" (UniqueName: \"kubernetes.io/projected/174be2db-9c13-45fe-b91f-c3302ecb57e2-kube-api-access-jjjjr\") pod \"node-resolver-8wd8v\" (UID: \"174be2db-9c13-45fe-b91f-c3302ecb57e2\") " pod="openshift-dns/node-resolver-8wd8v" Oct 30 01:28:21.327659 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327635 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-var-lib-cni-multus\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.328306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327673 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-kubelet-config\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:21.328306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327693 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f486e9e-aa32-4a9a-877b-eca8dbba7091-host\") pod \"node-ca-lq7lj\" (UID: \"4f486e9e-aa32-4a9a-877b-eca8dbba7091\") " pod="openshift-image-registry/node-ca-lq7lj" Oct 30 01:28:21.328306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327710 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwgtk\" (UniqueName: \"kubernetes.io/projected/6ff44b73-9f64-4fca-95f8-6a5a24540843-kube-api-access-kwgtk\") pod \"network-metrics-daemon-plsqk\" (UID: \"6ff44b73-9f64-4fca-95f8-6a5a24540843\") " pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:21.328306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327772 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c01f6b4a-5666-4734-b403-3818692c3a74-ovn-node-metrics-cert\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.328306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327803 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkjlt\" (UniqueName: \"kubernetes.io/projected/c01f6b4a-5666-4734-b403-3818692c3a74-kube-api-access-kkjlt\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.328306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327828 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-lib-modules\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.328306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327862 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4-iptables-alerter-script\") pod \"iptables-alerter-p9b7v\" (UID: \"07c8420b-05bf-4d3e-a1a2-7a6d55fffad4\") " pod="openshift-network-operator/iptables-alerter-p9b7v" Oct 30 01:28:21.328306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327885 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-cni-binary-copy\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.328306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327941 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbxj6\" (UniqueName: \"kubernetes.io/projected/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-kube-api-access-dbxj6\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.328306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.327986 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs\") pod \"network-metrics-daemon-plsqk\" (UID: \"6ff44b73-9f64-4fca-95f8-6a5a24540843\") " pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:21.328306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328018 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-systemd-units\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.328306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328040 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-run-openvswitch\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.328306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328073 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-sys\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.328306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328122 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/174be2db-9c13-45fe-b91f-c3302ecb57e2-hosts-file\") pod \"node-resolver-8wd8v\" (UID: \"174be2db-9c13-45fe-b91f-c3302ecb57e2\") " pod="openshift-dns/node-resolver-8wd8v" Oct 30 01:28:21.328306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328149 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4-host-slash\") pod \"iptables-alerter-p9b7v\" (UID: \"07c8420b-05bf-4d3e-a1a2-7a6d55fffad4\") " pod="openshift-network-operator/iptables-alerter-p9b7v" Oct 30 01:28:21.328306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328175 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-run-k8s-cni-cncf-io\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.329058 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328201 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-run-netns\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.329058 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328216 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-var-lib-kubelet\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.329058 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328231 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-log-socket\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.329058 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328252 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-systemd\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.329058 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328299 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgnfn\" (UniqueName: \"kubernetes.io/projected/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4-kube-api-access-qgnfn\") pod \"iptables-alerter-p9b7v\" (UID: \"07c8420b-05bf-4d3e-a1a2-7a6d55fffad4\") " pod="openshift-network-operator/iptables-alerter-p9b7v" Oct 30 01:28:21.329058 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328315 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-cnibin\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.329058 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328335 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-daemon-config\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.329058 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328359 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40d2d25c-e7a0-4ded-9674-19c6d371522e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.329058 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328378 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-run-systemd\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.329058 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328392 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-cni-bin\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.329058 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328412 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c01f6b4a-5666-4734-b403-3818692c3a74-ovnkube-script-lib\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.329058 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328435 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-sysconfig\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.329058 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328463 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f486e9e-aa32-4a9a-877b-eca8dbba7091-serviceca\") pod \"node-ca-lq7lj\" (UID: \"4f486e9e-aa32-4a9a-877b-eca8dbba7091\") " pod="openshift-image-registry/node-ca-lq7lj" Oct 30 01:28:21.329058 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328478 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-var-lib-cni-bin\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.329058 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328506 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.329058 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328525 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:21.329677 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328543 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-sysctl-conf\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.329677 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328557 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.329677 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328570 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-registration-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.329677 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328584 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-socket-dir-parent\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.329677 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328610 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-run-multus-certs\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.329677 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328635 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40d2d25c-e7a0-4ded-9674-19c6d371522e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.329677 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328656 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-run-ovn-kubernetes\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.329677 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328670 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174be2db-9c13-45fe-b91f-c3302ecb57e2-tmp-dir\") pod \"node-resolver-8wd8v\" (UID: \"174be2db-9c13-45fe-b91f-c3302ecb57e2\") " pod="openshift-dns/node-resolver-8wd8v" Oct 30 01:28:21.329677 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328686 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqrp9\" (UniqueName: \"kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9\") pod \"network-check-target-pct74\" (UID: \"8af570c7-62ff-4f33-93df-1714b5ffc20c\") " pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:21.329677 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328700 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40d2d25c-e7a0-4ded-9674-19c6d371522e-system-cni-dir\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.329677 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328728 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40d2d25c-e7a0-4ded-9674-19c6d371522e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.329677 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328765 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-host\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.329677 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328780 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-socket-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.329677 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328795 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwl72\" (UniqueName: \"kubernetes.io/projected/d50883a3-5056-4826-884e-90fe75441e5d-kube-api-access-gwl72\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.329677 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328813 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-system-cni-dir\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.329677 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328827 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-cni-dir\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.330306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328842 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40d2d25c-e7a0-4ded-9674-19c6d371522e-cnibin\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.330306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328864 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-var-lib-openvswitch\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.330306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328885 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-var-lib-kubelet\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.330306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328901 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-kubelet\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.330306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328914 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-run-ovn\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.330306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328931 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c01f6b4a-5666-4734-b403-3818692c3a74-env-overrides\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.330306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328957 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.330306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.328972 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-hostroot\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.330306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.329008 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-node-log\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.330306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.329046 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0d9a754-4e82-4dd5-b423-50624589bdda-agent-certs\") pod \"konnectivity-agent-44ftt\" (UID: \"a0d9a754-4e82-4dd5-b423-50624589bdda\") " pod="kube-system/konnectivity-agent-44ftt" Oct 30 01:28:21.330306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.329063 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-kubernetes\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.330306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.329078 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40d2d25c-e7a0-4ded-9674-19c6d371522e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.330306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.329112 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-modprobe-d\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.330306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.329127 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtwhw\" (UniqueName: \"kubernetes.io/projected/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-kube-api-access-vtwhw\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.363319 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.363260 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 01:28:21.363319 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.363293 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2gfdm\"" Oct 30 01:28:21.363319 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.363270 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 01:28:21.363647 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.363617 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 01:28:21.363647 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.363619 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 01:28:21.363794 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.363739 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xglfc\"" Oct 30 01:28:21.429900 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.429871 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-systemd\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.430072 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.429910 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgnfn\" (UniqueName: \"kubernetes.io/projected/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4-kube-api-access-qgnfn\") pod \"iptables-alerter-p9b7v\" (UID: \"07c8420b-05bf-4d3e-a1a2-7a6d55fffad4\") " pod="openshift-network-operator/iptables-alerter-p9b7v" Oct 30 01:28:21.430072 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.429933 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-cnibin\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.430072 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430022 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-cnibin\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.430072 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430031 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-systemd\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.430302 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430072 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-daemon-config\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.430302 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430114 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40d2d25c-e7a0-4ded-9674-19c6d371522e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.430302 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430140 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-run-systemd\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.430302 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430165 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-cni-bin\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.430302 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430189 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c01f6b4a-5666-4734-b403-3818692c3a74-ovnkube-script-lib\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.430302 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430212 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-sysconfig\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.430302 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430235 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f486e9e-aa32-4a9a-877b-eca8dbba7091-serviceca\") pod \"node-ca-lq7lj\" (UID: \"4f486e9e-aa32-4a9a-877b-eca8dbba7091\") " pod="openshift-image-registry/node-ca-lq7lj" Oct 30 01:28:21.430302 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430259 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-var-lib-cni-bin\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.430302 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430286 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.430732 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430310 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:21.430732 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430336 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-sysctl-conf\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.430732 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430370 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.430732 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430406 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-registration-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.430732 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430440 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-socket-dir-parent\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.430732 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430466 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-run-multus-certs\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.430732 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430492 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40d2d25c-e7a0-4ded-9674-19c6d371522e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.430732 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430516 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-run-ovn-kubernetes\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.430732 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430541 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174be2db-9c13-45fe-b91f-c3302ecb57e2-tmp-dir\") pod \"node-resolver-8wd8v\" (UID: \"174be2db-9c13-45fe-b91f-c3302ecb57e2\") " pod="openshift-dns/node-resolver-8wd8v" Oct 30 01:28:21.430732 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430564 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqrp9\" (UniqueName: \"kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9\") pod \"network-check-target-pct74\" (UID: \"8af570c7-62ff-4f33-93df-1714b5ffc20c\") " pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:21.430732 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430592 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40d2d25c-e7a0-4ded-9674-19c6d371522e-system-cni-dir\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.430732 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430618 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40d2d25c-e7a0-4ded-9674-19c6d371522e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.430732 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430642 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-host\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.430732 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430664 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-socket-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.430732 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430694 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwl72\" (UniqueName: \"kubernetes.io/projected/d50883a3-5056-4826-884e-90fe75441e5d-kube-api-access-gwl72\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.430732 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430719 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-system-cni-dir\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.431475 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430742 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-cni-dir\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.431475 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430767 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40d2d25c-e7a0-4ded-9674-19c6d371522e-cnibin\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.431475 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430804 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-var-lib-openvswitch\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.431475 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430831 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-var-lib-kubelet\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.431475 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430849 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c01f6b4a-5666-4734-b403-3818692c3a74-ovnkube-script-lib\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.431475 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430853 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-kubelet\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.431475 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430895 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-kubelet\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.431475 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430906 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-run-ovn\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.431475 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430935 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-run-ovn\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.431475 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430954 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-sysconfig\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.431475 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.430980 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c01f6b4a-5666-4734-b403-3818692c3a74-env-overrides\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.431475 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431005 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.431475 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431031 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-hostroot\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.431475 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431054 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-node-log\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.431475 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431078 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0d9a754-4e82-4dd5-b423-50624589bdda-agent-certs\") pod \"konnectivity-agent-44ftt\" (UID: \"a0d9a754-4e82-4dd5-b423-50624589bdda\") " pod="kube-system/konnectivity-agent-44ftt" Oct 30 01:28:21.431475 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431120 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-kubernetes\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.431475 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431144 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40d2d25c-e7a0-4ded-9674-19c6d371522e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.431475 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431183 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-run-systemd\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.432329 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431200 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-cni-bin\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.432329 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431270 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-registration-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.432329 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431299 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-modprobe-d\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.432329 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431190 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-modprobe-d\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.432329 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431359 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-host\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.432329 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431360 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtwhw\" (UniqueName: \"kubernetes.io/projected/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-kube-api-access-vtwhw\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.432329 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431392 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-cni-netd\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.432329 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431407 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-var-lib-cni-bin\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.432329 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431419 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0d9a754-4e82-4dd5-b423-50624589bdda-konnectivity-ca\") pod \"konnectivity-agent-44ftt\" (UID: \"a0d9a754-4e82-4dd5-b423-50624589bdda\") " pod="kube-system/konnectivity-agent-44ftt" Oct 30 01:28:21.432329 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:21.431452 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:21.432329 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431463 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-tmp\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.432329 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431498 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwzmn\" (UniqueName: \"kubernetes.io/projected/40d2d25c-e7a0-4ded-9674-19c6d371522e-kube-api-access-qwzmn\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.432329 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:21.431519 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret podName:e0bbf87d-7baa-4b99-9b80-695c2c99c4c8 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:21.931502663 +0000 UTC m=+3.068378765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret") pod "global-pull-secret-syncer-55kfb" (UID: "e0bbf87d-7baa-4b99-9b80-695c2c99c4c8") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:21.432329 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431543 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-etc-openvswitch\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.432329 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431563 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c01f6b4a-5666-4734-b403-3818692c3a74-ovnkube-config\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.432329 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431580 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-dbus\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:21.432329 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431713 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-socket-dir-parent\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.433126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431727 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-dbus\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:21.433126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431763 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.433126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431789 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-run-multus-certs\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.433126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431895 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-etc-openvswitch\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.433126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432238 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-run-ovn-kubernetes\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.433126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.431621 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-device-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.433126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432296 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-sys-fs\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.433126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432307 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40d2d25c-e7a0-4ded-9674-19c6d371522e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.433126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432322 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvtjv\" (UniqueName: \"kubernetes.io/projected/4f486e9e-aa32-4a9a-877b-eca8dbba7091-kube-api-access-wvtjv\") pod \"node-ca-lq7lj\" (UID: \"4f486e9e-aa32-4a9a-877b-eca8dbba7091\") " pod="openshift-image-registry/node-ca-lq7lj" Oct 30 01:28:21.433126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432350 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-etc-kubernetes\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.433126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432373 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-run\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.433126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432381 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-socket-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.433126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432401 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-conf-dir\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.433126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432421 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-slash\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.433126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432442 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-run-netns\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.433126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432470 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-sysctl-d\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.433126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432476 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-sysctl-conf\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.433823 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432494 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-tuned\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.433823 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432518 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjjjr\" (UniqueName: \"kubernetes.io/projected/174be2db-9c13-45fe-b91f-c3302ecb57e2-kube-api-access-jjjjr\") pod \"node-resolver-8wd8v\" (UID: \"174be2db-9c13-45fe-b91f-c3302ecb57e2\") " pod="openshift-dns/node-resolver-8wd8v" Oct 30 01:28:21.433823 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432537 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.433823 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432549 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174be2db-9c13-45fe-b91f-c3302ecb57e2-tmp-dir\") pod \"node-resolver-8wd8v\" (UID: \"174be2db-9c13-45fe-b91f-c3302ecb57e2\") " pod="openshift-dns/node-resolver-8wd8v" Oct 30 01:28:21.433823 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432587 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-run\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.433823 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432596 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-system-cni-dir\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.433823 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432617 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-device-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.433823 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432647 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-cni-dir\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.433823 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432311 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c01f6b4a-5666-4734-b403-3818692c3a74-ovnkube-config\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.433823 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432646 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c01f6b4a-5666-4734-b403-3818692c3a74-env-overrides\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.433823 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432660 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40d2d25c-e7a0-4ded-9674-19c6d371522e-system-cni-dir\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.433823 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432673 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-sys-fs\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.433823 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432683 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40d2d25c-e7a0-4ded-9674-19c6d371522e-cnibin\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.433823 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432709 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-var-lib-cni-multus\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.433823 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432727 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-kubelet-config\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:21.433823 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432742 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f486e9e-aa32-4a9a-877b-eca8dbba7091-host\") pod \"node-ca-lq7lj\" (UID: \"4f486e9e-aa32-4a9a-877b-eca8dbba7091\") " pod="openshift-image-registry/node-ca-lq7lj" Oct 30 01:28:21.433823 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432751 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-conf-dir\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.433823 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432763 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwgtk\" (UniqueName: \"kubernetes.io/projected/6ff44b73-9f64-4fca-95f8-6a5a24540843-kube-api-access-kwgtk\") pod \"network-metrics-daemon-plsqk\" (UID: \"6ff44b73-9f64-4fca-95f8-6a5a24540843\") " pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:21.434506 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432780 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c01f6b4a-5666-4734-b403-3818692c3a74-ovn-node-metrics-cert\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.434506 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432794 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-slash\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.434506 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432796 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjlt\" (UniqueName: \"kubernetes.io/projected/c01f6b4a-5666-4734-b403-3818692c3a74-kube-api-access-kkjlt\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.434506 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432833 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-var-lib-cni-multus\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.434506 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432835 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-lib-modules\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.434506 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432856 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-etc-kubernetes\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.434506 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432862 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-kubelet-config\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:21.434506 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432863 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4-iptables-alerter-script\") pod \"iptables-alerter-p9b7v\" (UID: \"07c8420b-05bf-4d3e-a1a2-7a6d55fffad4\") " pod="openshift-network-operator/iptables-alerter-p9b7v" Oct 30 01:28:21.434506 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432890 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-cni-binary-copy\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.434506 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432905 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-var-lib-kubelet\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.434506 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432914 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbxj6\" (UniqueName: \"kubernetes.io/projected/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-kube-api-access-dbxj6\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.434506 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432931 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f486e9e-aa32-4a9a-877b-eca8dbba7091-host\") pod \"node-ca-lq7lj\" (UID: \"4f486e9e-aa32-4a9a-877b-eca8dbba7091\") " pod="openshift-image-registry/node-ca-lq7lj" Oct 30 01:28:21.434506 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432939 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs\") pod \"network-metrics-daemon-plsqk\" (UID: \"6ff44b73-9f64-4fca-95f8-6a5a24540843\") " pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:21.434506 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432963 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-systemd-units\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.434506 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432988 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-run-openvswitch\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.434506 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433012 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-sys\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.434506 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433018 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-var-lib-openvswitch\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.434506 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433038 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/174be2db-9c13-45fe-b91f-c3302ecb57e2-hosts-file\") pod \"node-resolver-8wd8v\" (UID: \"174be2db-9c13-45fe-b91f-c3302ecb57e2\") " pod="openshift-dns/node-resolver-8wd8v" Oct 30 01:28:21.435265 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433061 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4-host-slash\") pod \"iptables-alerter-p9b7v\" (UID: \"07c8420b-05bf-4d3e-a1a2-7a6d55fffad4\") " pod="openshift-network-operator/iptables-alerter-p9b7v" Oct 30 01:28:21.435265 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-run-k8s-cni-cncf-io\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.435265 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433110 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.435265 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433131 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-run-netns\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.435265 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433151 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-hostroot\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.435265 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433153 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-var-lib-kubelet\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.435265 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433192 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-log-socket\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.435265 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433209 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40d2d25c-e7a0-4ded-9674-19c6d371522e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.435265 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433226 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-run-netns\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.435265 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433192 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-var-lib-kubelet\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.435265 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433331 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-sysctl-d\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.435265 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433345 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-run-openvswitch\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.435265 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.432987 2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 01:28:21.435265 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433378 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-node-log\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.435265 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433405 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-log-socket\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.435265 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433446 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-lib-modules\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.435265 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433452 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4-host-slash\") pod \"iptables-alerter-p9b7v\" (UID: \"07c8420b-05bf-4d3e-a1a2-7a6d55fffad4\") " pod="openshift-network-operator/iptables-alerter-p9b7v" Oct 30 01:28:21.435265 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433504 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/174be2db-9c13-45fe-b91f-c3302ecb57e2-hosts-file\") pod \"node-resolver-8wd8v\" (UID: \"174be2db-9c13-45fe-b91f-c3302ecb57e2\") " pod="openshift-dns/node-resolver-8wd8v" Oct 30 01:28:21.436161 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433505 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-sys\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.436161 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433545 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-run-netns\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.436161 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433573 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-run-k8s-cni-cncf-io\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.436161 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433603 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-kubernetes\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.436161 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433609 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-cni-netd\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.436161 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:21.433624 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:21.436161 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433652 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-systemd-units\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.436161 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:21.433672 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs podName:6ff44b73-9f64-4fca-95f8-6a5a24540843 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:21.933656259 +0000 UTC m=+3.070532349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs") pod "network-metrics-daemon-plsqk" (UID: "6ff44b73-9f64-4fca-95f8-6a5a24540843") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:21.436161 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.433797 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-daemon-config\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.436161 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.434061 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-cni-binary-copy\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.436161 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.434251 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f486e9e-aa32-4a9a-877b-eca8dbba7091-serviceca\") pod \"node-ca-lq7lj\" (UID: \"4f486e9e-aa32-4a9a-877b-eca8dbba7091\") " pod="openshift-image-registry/node-ca-lq7lj" Oct 30 01:28:21.436161 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.434658 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0d9a754-4e82-4dd5-b423-50624589bdda-konnectivity-ca\") pod \"konnectivity-agent-44ftt\" (UID: \"a0d9a754-4e82-4dd5-b423-50624589bdda\") " pod="kube-system/konnectivity-agent-44ftt" Oct 30 01:28:21.436161 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.435430 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40d2d25c-e7a0-4ded-9674-19c6d371522e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.436161 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.435452 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4-iptables-alerter-script\") pod \"iptables-alerter-p9b7v\" (UID: \"07c8420b-05bf-4d3e-a1a2-7a6d55fffad4\") " pod="openshift-network-operator/iptables-alerter-p9b7v" Oct 30 01:28:21.436161 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.435783 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40d2d25c-e7a0-4ded-9674-19c6d371522e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.437663 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.437637 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-tuned\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.437808 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.437787 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c01f6b4a-5666-4734-b403-3818692c3a74-ovn-node-metrics-cert\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.438066 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.438050 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0d9a754-4e82-4dd5-b423-50624589bdda-agent-certs\") pod \"konnectivity-agent-44ftt\" (UID: \"a0d9a754-4e82-4dd5-b423-50624589bdda\") " pod="kube-system/konnectivity-agent-44ftt" Oct 30 01:28:21.439462 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.439419 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-tmp\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.460214 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.460179 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgnfn\" (UniqueName: \"kubernetes.io/projected/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4-kube-api-access-qgnfn\") pod \"iptables-alerter-p9b7v\" (UID: \"07c8420b-05bf-4d3e-a1a2-7a6d55fffad4\") " pod="openshift-network-operator/iptables-alerter-p9b7v" Oct 30 01:28:21.475441 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.475413 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwzmn\" (UniqueName: \"kubernetes.io/projected/40d2d25c-e7a0-4ded-9674-19c6d371522e-kube-api-access-qwzmn\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.480226 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:21.480204 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:21.480226 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:21.480229 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:21.480381 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:21.480243 2582 projected.go:196] Error preparing data for projected volume kube-api-access-xqrp9 for pod openshift-network-diagnostics/network-check-target-pct74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:21.480381 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:21.480301 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9 podName:8af570c7-62ff-4f33-93df-1714b5ffc20c nodeName:}" failed. No retries permitted until 2026-10-30 01:28:21.980287275 +0000 UTC m=+3.117163362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xqrp9" (UniqueName: "kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9") pod "network-check-target-pct74" (UID: "8af570c7-62ff-4f33-93df-1714b5ffc20c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:21.482747 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.482722 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvtjv\" (UniqueName: \"kubernetes.io/projected/4f486e9e-aa32-4a9a-877b-eca8dbba7091-kube-api-access-wvtjv\") pod \"node-ca-lq7lj\" (UID: \"4f486e9e-aa32-4a9a-877b-eca8dbba7091\") " pod="openshift-image-registry/node-ca-lq7lj" Oct 30 01:28:21.482747 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.482732 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwgtk\" (UniqueName: \"kubernetes.io/projected/6ff44b73-9f64-4fca-95f8-6a5a24540843-kube-api-access-kwgtk\") pod \"network-metrics-daemon-plsqk\" (UID: \"6ff44b73-9f64-4fca-95f8-6a5a24540843\") " pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:21.490586 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.488562 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtwhw\" (UniqueName: \"kubernetes.io/projected/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-kube-api-access-vtwhw\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.493763 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.493733 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbxj6\" (UniqueName: \"kubernetes.io/projected/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-kube-api-access-dbxj6\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.493966 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.493938 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkjlt\" (UniqueName: \"kubernetes.io/projected/c01f6b4a-5666-4734-b403-3818692c3a74-kube-api-access-kkjlt\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.494390 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.494365 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjjjr\" (UniqueName: \"kubernetes.io/projected/174be2db-9c13-45fe-b91f-c3302ecb57e2-kube-api-access-jjjjr\") pod \"node-resolver-8wd8v\" (UID: \"174be2db-9c13-45fe-b91f-c3302ecb57e2\") " pod="openshift-dns/node-resolver-8wd8v" Oct 30 01:28:21.495789 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.495763 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwl72\" (UniqueName: \"kubernetes.io/projected/d50883a3-5056-4826-884e-90fe75441e5d-kube-api-access-gwl72\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.535190 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.535158 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:28:21.545655 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.545627 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8wd8v" Oct 30 01:28:21.546305 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:21.546274 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1facbe7_deb0_4fec_94de_42a0dd91ff4f.slice/crio-523df0c5ecd5e58cd0c7df22bff88e79fe2b97f1b39a4b430606e046a598dd9f WatchSource:0}: Error finding container 523df0c5ecd5e58cd0c7df22bff88e79fe2b97f1b39a4b430606e046a598dd9f: Status 404 returned error can't find the container with id 523df0c5ecd5e58cd0c7df22bff88e79fe2b97f1b39a4b430606e046a598dd9f Oct 30 01:28:21.555009 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:21.554982 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174be2db_9c13_45fe_b91f_c3302ecb57e2.slice/crio-15a65d279b86c4bb5028f6df9dd3f842f2fe926c1454b8656321728f4455f3b4 WatchSource:0}: Error finding container 15a65d279b86c4bb5028f6df9dd3f842f2fe926c1454b8656321728f4455f3b4: Status 404 returned error can't find the container with id 15a65d279b86c4bb5028f6df9dd3f842f2fe926c1454b8656321728f4455f3b4 Oct 30 01:28:21.558716 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.558697 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:28:21.561849 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.561827 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p9b7v" Oct 30 01:28:21.569436 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:21.569398 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50883a3_5056_4826_884e_90fe75441e5d.slice/crio-8b47ad09bf1c892f8f126aba288c1e4c57b17c01d77a7fff7850ee48b39bf2ce WatchSource:0}: Error finding container 8b47ad09bf1c892f8f126aba288c1e4c57b17c01d77a7fff7850ee48b39bf2ce: Status 404 returned error can't find the container with id 8b47ad09bf1c892f8f126aba288c1e4c57b17c01d77a7fff7850ee48b39bf2ce Oct 30 01:28:21.583420 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.583396 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lq7lj" Oct 30 01:28:21.589486 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.589469 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sgm2w" Oct 30 01:28:21.591653 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.591632 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:21.594312 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:21.594277 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f486e9e_aa32_4a9a_877b_eca8dbba7091.slice/crio-3daab102c6828b9bfd879686ef3bc503036ede472096f62ebe89b7894bf61417 WatchSource:0}: Error finding container 3daab102c6828b9bfd879686ef3bc503036ede472096f62ebe89b7894bf61417: Status 404 returned error can't find the container with id 3daab102c6828b9bfd879686ef3bc503036ede472096f62ebe89b7894bf61417 Oct 30 01:28:21.602789 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:21.602767 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01f6b4a_5666_4734_b403_3818692c3a74.slice/crio-7f1b5cde92f5e98099f7b8b4e3ce0aa5a8f19a3bdacfa18e601adbc56d9ca63b WatchSource:0}: Error finding container 7f1b5cde92f5e98099f7b8b4e3ce0aa5a8f19a3bdacfa18e601adbc56d9ca63b: Status 404 returned error can't find the container with id 7f1b5cde92f5e98099f7b8b4e3ce0aa5a8f19a3bdacfa18e601adbc56d9ca63b Oct 30 01:28:21.604202 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.604184 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-44ftt" Oct 30 01:28:21.608893 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.608872 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:28:21.616463 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:21.616434 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d9a754_4e82_4dd5_b423_50624589bdda.slice/crio-3f185188f4a22d5bd7e0b9164ff1701202c2eef5e35dfbf8fd8d004879e97812 WatchSource:0}: Error finding container 3f185188f4a22d5bd7e0b9164ff1701202c2eef5e35dfbf8fd8d004879e97812: Status 404 returned error can't find the container with id 3f185188f4a22d5bd7e0b9164ff1701202c2eef5e35dfbf8fd8d004879e97812 Oct 30 01:28:21.621918 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:28:21.621810 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d2d25c_e7a0_4ded_9674_19c6d371522e.slice/crio-2d9c51ec0745adf7780cd055fbbbcf067c6bc5a9a4f987ec6e3ead8258ce72ca WatchSource:0}: Error finding container 2d9c51ec0745adf7780cd055fbbbcf067c6bc5a9a4f987ec6e3ead8258ce72ca: Status 404 returned error can't find the container with id 2d9c51ec0745adf7780cd055fbbbcf067c6bc5a9a4f987ec6e3ead8258ce72ca Oct 30 01:28:21.938238 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.938204 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs\") pod \"network-metrics-daemon-plsqk\" (UID: \"6ff44b73-9f64-4fca-95f8-6a5a24540843\") " pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:21.938394 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:21.938261 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:21.938394 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:21.938380 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:21.938394 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:21.938390 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:21.938599 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:21.938457 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs podName:6ff44b73-9f64-4fca-95f8-6a5a24540843 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:22.938437334 +0000 UTC m=+4.075313422 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs") pod "network-metrics-daemon-plsqk" (UID: "6ff44b73-9f64-4fca-95f8-6a5a24540843") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:21.938599 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:21.938479 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret podName:e0bbf87d-7baa-4b99-9b80-695c2c99c4c8 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:22.938468622 +0000 UTC m=+4.075344736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret") pod "global-pull-secret-syncer-55kfb" (UID: "e0bbf87d-7baa-4b99-9b80-695c2c99c4c8") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:22.039988 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.039579 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqrp9\" (UniqueName: \"kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9\") pod \"network-check-target-pct74\" (UID: \"8af570c7-62ff-4f33-93df-1714b5ffc20c\") " pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:22.039988 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:22.039738 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:22.039988 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:22.039761 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:22.039988 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:22.039774 2582 projected.go:196] Error preparing data for projected volume kube-api-access-xqrp9 for pod openshift-network-diagnostics/network-check-target-pct74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:22.039988 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:22.039830 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9 podName:8af570c7-62ff-4f33-93df-1714b5ffc20c nodeName:}" failed. No retries permitted until 2026-10-30 01:28:23.039812389 +0000 UTC m=+4.176688463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqrp9" (UniqueName: "kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9") pod "network-check-target-pct74" (UID: "8af570c7-62ff-4f33-93df-1714b5ffc20c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:22.311173 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.311071 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 01:23:21 +0000 UTC" deadline="2028-07-02 09:58:09.27212001 +0000 UTC" Oct 30 01:28:22.311173 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.311150 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14672h29m46.960974638s" Oct 30 01:28:22.468039 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.467122 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:22.468039 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:22.467502 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:22.468039 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.467902 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:22.468039 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:22.467993 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:22.488240 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.488170 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" event={"ID":"d1facbe7-deb0-4fec-94de-42a0dd91ff4f","Type":"ContainerStarted","Data":"523df0c5ecd5e58cd0c7df22bff88e79fe2b97f1b39a4b430606e046a598dd9f"} Oct 30 01:28:22.494762 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.494729 2582 generic.go:356] "Generic (PLEG): container finished" podID="f54d525f1aa0b0582cbd13c31e87b9a6" containerID="e9b9d416301b94032e01caf0dc6ea22884019e88a4b2de8af8df180527542e45" exitCode=0 Oct 30 01:28:22.494889 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.494836 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" event={"ID":"f54d525f1aa0b0582cbd13c31e87b9a6","Type":"ContainerDied","Data":"e9b9d416301b94032e01caf0dc6ea22884019e88a4b2de8af8df180527542e45"} Oct 30 01:28:22.506355 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.506314 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zrv6d" event={"ID":"40d2d25c-e7a0-4ded-9674-19c6d371522e","Type":"ContainerStarted","Data":"2d9c51ec0745adf7780cd055fbbbcf067c6bc5a9a4f987ec6e3ead8258ce72ca"} Oct 30 01:28:22.514215 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.514168 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-44ftt" event={"ID":"a0d9a754-4e82-4dd5-b423-50624589bdda","Type":"ContainerStarted","Data":"3f185188f4a22d5bd7e0b9164ff1701202c2eef5e35dfbf8fd8d004879e97812"} Oct 30 01:28:22.527665 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.527637 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sgm2w" event={"ID":"18cee84e-69dd-4296-98d7-de2e6e0c6fe0","Type":"ContainerStarted","Data":"73d90759d94928512bd548cf432ea46ffb8eab3daea096ef0d84adb8764c6ac8"} Oct 30 01:28:22.531412 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.531352 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p9b7v" event={"ID":"07c8420b-05bf-4d3e-a1a2-7a6d55fffad4","Type":"ContainerStarted","Data":"1b9a49f978b8703acca46ee6d0b3da0bcffee1ff4f686b59538aeac1d97ab664"} Oct 30 01:28:22.553615 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.552820 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal" event={"ID":"8da95128540617dd483ff23c5d0ddd95","Type":"ContainerStarted","Data":"92bc8c9e67f63521f31b6ee495a6da4a2cfab35ce71e9e29364344f4a272ca7f"} Oct 30 01:28:22.559332 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.559214 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" event={"ID":"c01f6b4a-5666-4734-b403-3818692c3a74","Type":"ContainerStarted","Data":"7f1b5cde92f5e98099f7b8b4e3ce0aa5a8f19a3bdacfa18e601adbc56d9ca63b"} Oct 30 01:28:22.569193 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.569113 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lq7lj" event={"ID":"4f486e9e-aa32-4a9a-877b-eca8dbba7091","Type":"ContainerStarted","Data":"3daab102c6828b9bfd879686ef3bc503036ede472096f62ebe89b7894bf61417"} Oct 30 01:28:22.575037 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.574969 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" event={"ID":"d50883a3-5056-4826-884e-90fe75441e5d","Type":"ContainerStarted","Data":"8b47ad09bf1c892f8f126aba288c1e4c57b17c01d77a7fff7850ee48b39bf2ce"} Oct 30 01:28:22.579678 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.579620 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8wd8v" event={"ID":"174be2db-9c13-45fe-b91f-c3302ecb57e2","Type":"ContainerStarted","Data":"15a65d279b86c4bb5028f6df9dd3f842f2fe926c1454b8656321728f4455f3b4"} Oct 30 01:28:22.954196 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.954132 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs\") pod \"network-metrics-daemon-plsqk\" (UID: \"6ff44b73-9f64-4fca-95f8-6a5a24540843\") " pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:22.954196 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:22.954198 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:22.954526 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:22.954473 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:22.954578 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:22.954538 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs podName:6ff44b73-9f64-4fca-95f8-6a5a24540843 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:24.95451881 +0000 UTC m=+6.091394901 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs") pod "network-metrics-daemon-plsqk" (UID: "6ff44b73-9f64-4fca-95f8-6a5a24540843") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:22.954934 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:22.954914 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:22.955017 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:22.954969 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret podName:e0bbf87d-7baa-4b99-9b80-695c2c99c4c8 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:24.954955609 +0000 UTC m=+6.091831698 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret") pod "global-pull-secret-syncer-55kfb" (UID: "e0bbf87d-7baa-4b99-9b80-695c2c99c4c8") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:23.055726 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:23.055027 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqrp9\" (UniqueName: \"kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9\") pod \"network-check-target-pct74\" (UID: \"8af570c7-62ff-4f33-93df-1714b5ffc20c\") " pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:23.055726 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:23.055192 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:23.055726 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:23.055210 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:23.055726 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:23.055223 2582 projected.go:196] Error preparing data for projected volume kube-api-access-xqrp9 for pod openshift-network-diagnostics/network-check-target-pct74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:23.055726 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:23.055282 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9 podName:8af570c7-62ff-4f33-93df-1714b5ffc20c nodeName:}" failed. No retries permitted until 2026-10-30 01:28:25.055263028 +0000 UTC m=+6.192139118 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqrp9" (UniqueName: "kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9") pod "network-check-target-pct74" (UID: "8af570c7-62ff-4f33-93df-1714b5ffc20c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:23.312223 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:23.312129 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 01:23:21 +0000 UTC" deadline="2028-04-30 12:08:29.493466262 +0000 UTC" Oct 30 01:28:23.312223 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:23.312170 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13162h40m6.181300594s" Oct 30 01:28:23.467246 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:23.467213 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:23.467416 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:23.467356 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:23.591917 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:23.591837 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" event={"ID":"f54d525f1aa0b0582cbd13c31e87b9a6","Type":"ContainerStarted","Data":"77dcb2287173250bf1a38b22c94c832988db660739035c57c87554bb54e194aa"} Oct 30 01:28:23.633487 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:23.633431 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" podStartSLOduration=3.633412474 podStartE2EDuration="3.633412474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:23.632797784 +0000 UTC m=+4.769673881" watchObservedRunningTime="2026-10-30 01:28:23.633412474 +0000 UTC m=+4.770288574" Oct 30 01:28:23.633661 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:23.633585 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal" podStartSLOduration=3.6335755880000002 podStartE2EDuration="3.633575588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:22.592425814 +0000 UTC m=+3.729301911" watchObservedRunningTime="2026-10-30 01:28:23.633575588 +0000 UTC m=+4.770451685" Oct 30 01:28:24.467220 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:24.467188 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:24.467613 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:24.467329 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:24.467613 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:24.467355 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:24.467613 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:24.467479 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:24.599474 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:24.599427 2582 generic.go:356] "Generic (PLEG): container finished" podID="40d2d25c-e7a0-4ded-9674-19c6d371522e" containerID="9672f4742ddd3de805c65ac08e69025fc12592a746f02c5d00785aa1e7f9d963" exitCode=0 Oct 30 01:28:24.600598 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:24.600567 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zrv6d" event={"ID":"40d2d25c-e7a0-4ded-9674-19c6d371522e","Type":"ContainerDied","Data":"9672f4742ddd3de805c65ac08e69025fc12592a746f02c5d00785aa1e7f9d963"} Oct 30 01:28:24.971670 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:24.971628 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:24.971862 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:24.971727 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs\") pod \"network-metrics-daemon-plsqk\" (UID: \"6ff44b73-9f64-4fca-95f8-6a5a24540843\") " pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:24.971862 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:24.971858 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:24.971971 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:24.971921 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs podName:6ff44b73-9f64-4fca-95f8-6a5a24540843 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:28.971901233 +0000 UTC m=+10.108777330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs") pod "network-metrics-daemon-plsqk" (UID: "6ff44b73-9f64-4fca-95f8-6a5a24540843") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:24.972030 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:24.971989 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:24.972030 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:24.972027 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret podName:e0bbf87d-7baa-4b99-9b80-695c2c99c4c8 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:28.972015647 +0000 UTC m=+10.108891741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret") pod "global-pull-secret-syncer-55kfb" (UID: "e0bbf87d-7baa-4b99-9b80-695c2c99c4c8") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:25.072668 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:25.072632 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqrp9\" (UniqueName: \"kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9\") pod \"network-check-target-pct74\" (UID: \"8af570c7-62ff-4f33-93df-1714b5ffc20c\") " pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:25.072959 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:25.072768 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:25.072959 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:25.072783 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:25.072959 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:25.072792 2582 projected.go:196] Error preparing data for projected volume kube-api-access-xqrp9 for pod openshift-network-diagnostics/network-check-target-pct74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:25.072959 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:25.072835 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9 podName:8af570c7-62ff-4f33-93df-1714b5ffc20c nodeName:}" failed. No retries permitted until 2026-10-30 01:28:29.072823734 +0000 UTC m=+10.209699820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqrp9" (UniqueName: "kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9") pod "network-check-target-pct74" (UID: "8af570c7-62ff-4f33-93df-1714b5ffc20c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:25.468314 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:25.468279 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:25.468870 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:25.468448 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:26.467517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:26.467483 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:26.468149 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:26.467617 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:26.468149 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:26.467966 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:26.468149 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:26.468053 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:26.606472 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:26.605701 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-44ftt" event={"ID":"a0d9a754-4e82-4dd5-b423-50624589bdda","Type":"ContainerStarted","Data":"530364089ffaada12c6376852b7f3947d0a7c668f159c3a1ca44efab040a6ddd"} Oct 30 01:28:26.647975 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:26.647083 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-44ftt" podStartSLOduration=3.906514515 podStartE2EDuration="7.647065358s" totalImagesPullingTime="3.740550843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:26.646890363 +0000 UTC m=+7.783766461" watchObservedRunningTime="2026-10-30 01:28:26.647065358 +0000 UTC m=+7.783941454" Oct 30 01:28:27.469590 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:27.469107 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:27.469590 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:27.469248 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:27.616640 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:27.616547 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" event={"ID":"d50883a3-5056-4826-884e-90fe75441e5d","Type":"ContainerStarted","Data":"08a7cfb1de90c82952c0e9ebb936932f8dba95b5cc782f13202495e54cfc5c4a"} Oct 30 01:28:27.620019 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:27.619928 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lq7lj" event={"ID":"4f486e9e-aa32-4a9a-877b-eca8dbba7091","Type":"ContainerStarted","Data":"d67552ed4f4ce908f25cff03885d080119f5f21f83387c0758875d08df083c5b"} Oct 30 01:28:27.658878 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:27.658817 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lq7lj" podStartSLOduration=3.805520155 podStartE2EDuration="8.658797372s" totalImagesPullingTime="4.853277217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:27.6581878 +0000 UTC m=+8.795063897" watchObservedRunningTime="2026-10-30 01:28:27.658797372 +0000 UTC m=+8.795673469" Oct 30 01:28:28.467315 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:28.467271 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:28.467484 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:28.467429 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:28.467817 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:28.467785 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:28.467928 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:28.467877 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:28.876911 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:28.876886 2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 01:28:29.015334 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:29.015259 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs\") pod \"network-metrics-daemon-plsqk\" (UID: \"6ff44b73-9f64-4fca-95f8-6a5a24540843\") " pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:29.015334 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:29.015316 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:29.015546 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:29.015471 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:29.015546 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:29.015533 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret podName:e0bbf87d-7baa-4b99-9b80-695c2c99c4c8 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:37.015516658 +0000 UTC m=+18.152392734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret") pod "global-pull-secret-syncer-55kfb" (UID: "e0bbf87d-7baa-4b99-9b80-695c2c99c4c8") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:29.015974 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:29.015949 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:29.016083 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:29.016007 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs podName:6ff44b73-9f64-4fca-95f8-6a5a24540843 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:37.015992726 +0000 UTC m=+18.152868801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs") pod "network-metrics-daemon-plsqk" (UID: "6ff44b73-9f64-4fca-95f8-6a5a24540843") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:29.117050 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:29.116451 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqrp9\" (UniqueName: \"kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9\") pod \"network-check-target-pct74\" (UID: \"8af570c7-62ff-4f33-93df-1714b5ffc20c\") " pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:29.117050 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:29.116648 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:29.117050 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:29.116663 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:29.117050 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:29.116674 2582 projected.go:196] Error preparing data for projected volume kube-api-access-xqrp9 for pod openshift-network-diagnostics/network-check-target-pct74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:29.117050 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:29.116732 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9 podName:8af570c7-62ff-4f33-93df-1714b5ffc20c nodeName:}" failed. No retries permitted until 2026-10-30 01:28:37.116714493 +0000 UTC m=+18.253590591 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqrp9" (UniqueName: "kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9") pod "network-check-target-pct74" (UID: "8af570c7-62ff-4f33-93df-1714b5ffc20c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:29.415109 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:29.414987 2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T01:28:28.876906917Z","UUID":"32dbaac2-b938-448e-8843-7ec399ffbc01","Handler":null,"Name":"","Endpoint":""} Oct 30 01:28:29.416831 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:29.416810 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 01:28:29.416965 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:29.416839 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 01:28:29.468811 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:29.468162 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:29.468811 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:29.468271 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:29.627800 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:29.627759 2582 generic.go:356] "Generic (PLEG): container finished" podID="40d2d25c-e7a0-4ded-9674-19c6d371522e" containerID="60392adfc79717a9d92b7d8dcd47acd39253f2eba6f406c922d28ae707d5c4f6" exitCode=0 Oct 30 01:28:29.628425 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:29.628397 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zrv6d" event={"ID":"40d2d25c-e7a0-4ded-9674-19c6d371522e","Type":"ContainerDied","Data":"60392adfc79717a9d92b7d8dcd47acd39253f2eba6f406c922d28ae707d5c4f6"} Oct 30 01:28:29.631396 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:29.631369 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sgm2w" event={"ID":"18cee84e-69dd-4296-98d7-de2e6e0c6fe0","Type":"ContainerStarted","Data":"62d6898f77ee774b308cea3e7ca397b1ae167fdfa8e3d257656f5754dc5c80cb"} Oct 30 01:28:29.634619 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:29.634593 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" event={"ID":"d50883a3-5056-4826-884e-90fe75441e5d","Type":"ContainerStarted","Data":"7bc59c1fccd16b1f440e32a3b60a16f4cc551ad306b95d2f3bdfe9b21775e3c4"} Oct 30 01:28:30.374557 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:30.374485 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:30.468469 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:30.467859 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:30.468469 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:30.468008 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:30.468469 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:30.468343 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:30.468469 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:30.468433 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:30.568228 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:30.568195 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:30.638983 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:30.638918 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" event={"ID":"d50883a3-5056-4826-884e-90fe75441e5d","Type":"ContainerStarted","Data":"81edea99290d84143e9dfd4b7dd85f3e7a5ec953ca8a2dce988f0f96d1334774"} Oct 30 01:28:30.641214 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:30.641186 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8wd8v" event={"ID":"174be2db-9c13-45fe-b91f-c3302ecb57e2","Type":"ContainerStarted","Data":"d6dd49a9b8a5a8ec650277da4adb74d5a872837c5eb7189e41e74acbe6356660"} Oct 30 01:28:30.643434 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:30.643407 2582 generic.go:356] "Generic (PLEG): container finished" podID="40d2d25c-e7a0-4ded-9674-19c6d371522e" containerID="82a3e1615c34f568c7d52b6fad0d6f2ca4bd181b4bf7affe9af532f25d4c2e95" exitCode=0 Oct 30 01:28:30.643902 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:30.643882 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zrv6d" event={"ID":"40d2d25c-e7a0-4ded-9674-19c6d371522e","Type":"ContainerDied","Data":"82a3e1615c34f568c7d52b6fad0d6f2ca4bd181b4bf7affe9af532f25d4c2e95"} Oct 30 01:28:30.682672 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:30.682608 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-sgm2w" podStartSLOduration=4.582583482 podStartE2EDuration="11.682588124s" totalImagesPullingTime="7.100004642s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:29.723732161 +0000 UTC m=+10.860608257" watchObservedRunningTime="2026-10-30 01:28:30.682588124 +0000 UTC m=+11.819464220" Oct 30 01:28:30.705534 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:30.705499 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:30.725971 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:30.725930 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" podStartSLOduration=3.879083642 podStartE2EDuration="11.725914682s" totalImagesPullingTime="7.84683104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:30.68194013 +0000 UTC m=+11.818816227" watchObservedRunningTime="2026-10-30 01:28:30.725914682 +0000 UTC m=+11.862790780" Oct 30 01:28:30.726402 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:30.726372 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-8wd8v" podStartSLOduration=3.765057318 podStartE2EDuration="11.726363202s" totalImagesPullingTime="7.961305884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:30.725603398 +0000 UTC m=+11.862479495" watchObservedRunningTime="2026-10-30 01:28:30.726363202 +0000 UTC m=+11.863239297" Oct 30 01:28:30.914766 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:30.914583 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:31.032110 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:31.032048 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:31.149350 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:31.149294 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:31.437552 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:31.437464 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-44ftt" Oct 30 01:28:31.438154 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:31.438134 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-44ftt" Oct 30 01:28:31.467843 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:31.467463 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:31.467843 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:31.467561 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:31.651534 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:31.651505 2582 generic.go:356] "Generic (PLEG): container finished" podID="40d2d25c-e7a0-4ded-9674-19c6d371522e" containerID="8d0cd7fbfdf242a87ecff4368ad98d35d18422495cf3c8b10ee2a881dc8636f1" exitCode=0 Oct 30 01:28:31.651696 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:31.651593 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zrv6d" event={"ID":"40d2d25c-e7a0-4ded-9674-19c6d371522e","Type":"ContainerDied","Data":"8d0cd7fbfdf242a87ecff4368ad98d35d18422495cf3c8b10ee2a881dc8636f1"} Oct 30 01:28:31.653688 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:31.653638 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p9b7v" event={"ID":"07c8420b-05bf-4d3e-a1a2-7a6d55fffad4","Type":"ContainerStarted","Data":"32817e96b5075d9e9c51313807e07a5481fe89d42d441bbbe87a57628278bc68"} Oct 30 01:28:31.742073 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:31.741974 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-p9b7v" podStartSLOduration=5.12029999 podStartE2EDuration="12.741956682s" totalImagesPullingTime="7.621656692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:31.741619565 +0000 UTC m=+12.878495662" watchObservedRunningTime="2026-10-30 01:28:31.741956682 +0000 UTC m=+12.878832781" Oct 30 01:28:31.893148 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:31.893119 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:32.000639 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:32.000558 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:32.466793 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:32.466757 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:32.467408 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:32.466865 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:32.467408 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:32.466909 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:32.467408 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:32.467025 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:32.657982 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:32.657943 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" event={"ID":"d1facbe7-deb0-4fec-94de-42a0dd91ff4f","Type":"ContainerStarted","Data":"9f87c2a02dd2785a21439965d030701711c11cca4b6a5ba2b7848469a7eece0b"} Oct 30 01:28:32.701494 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:32.701435 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" podStartSLOduration=3.694585548 podStartE2EDuration="13.701417096s" totalImagesPullingTime="10.006831548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:32.700929671 +0000 UTC m=+13.837805766" watchObservedRunningTime="2026-10-30 01:28:32.701417096 +0000 UTC m=+13.838293210" Oct 30 01:28:32.809608 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:32.809512 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:32.913787 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:32.913756 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:33.214460 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:33.214427 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:33.324806 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:33.324739 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:33.467162 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:33.467071 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:33.467720 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:33.467253 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:33.664270 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:33.664233 2582 generic.go:356] "Generic (PLEG): container finished" podID="40d2d25c-e7a0-4ded-9674-19c6d371522e" containerID="23aefc04aa7988b03d3221c52dddd202e3592ec6c1666347ec04b658596083c6" exitCode=0 Oct 30 01:28:33.664433 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:33.664314 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zrv6d" event={"ID":"40d2d25c-e7a0-4ded-9674-19c6d371522e","Type":"ContainerDied","Data":"23aefc04aa7988b03d3221c52dddd202e3592ec6c1666347ec04b658596083c6"} Oct 30 01:28:33.908562 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:33.908528 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:34.013480 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:34.013411 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:34.467049 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:34.466838 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:34.467235 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:34.466912 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:34.467235 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:34.467181 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:34.467235 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:34.467225 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:34.670387 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:34.670353 2582 generic.go:356] "Generic (PLEG): container finished" podID="40d2d25c-e7a0-4ded-9674-19c6d371522e" containerID="ae0cf2c9763b957c33dc1285221b7d6b1b049258837608673d9044aa66f37c76" exitCode=0 Oct 30 01:28:34.670558 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:34.670399 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zrv6d" event={"ID":"40d2d25c-e7a0-4ded-9674-19c6d371522e","Type":"ContainerDied","Data":"ae0cf2c9763b957c33dc1285221b7d6b1b049258837608673d9044aa66f37c76"} Oct 30 01:28:35.467083 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:35.467046 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:35.467282 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:35.467172 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:35.675310 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:35.675275 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zrv6d" event={"ID":"40d2d25c-e7a0-4ded-9674-19c6d371522e","Type":"ContainerStarted","Data":"19b17fb6320a615c3ee8188894660ebc88187c0d125d3e07f6ecbfdd6fc5527a"} Oct 30 01:28:35.889074 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:35.889053 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:36.007690 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:36.007666 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:36.467032 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:36.467006 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:36.467235 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:36.467136 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:36.467235 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:36.467183 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:36.467617 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:36.467268 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:36.528345 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:36.528312 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:36.641952 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:36.641921 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:36.679981 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:36.679949 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" event={"ID":"c01f6b4a-5666-4734-b403-3818692c3a74","Type":"ContainerStarted","Data":"ab2bbe78753707239b537f30750d3a4d1414cf0675cb0d199396e37baea410e6"} Oct 30 01:28:36.680150 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:36.680028 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" event={"ID":"c01f6b4a-5666-4734-b403-3818692c3a74","Type":"ContainerStarted","Data":"f38a04f19d7b2a132948338ace182e8c2ada5455f62869604e1f7ee90834682f"} Oct 30 01:28:36.680150 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:36.680074 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" event={"ID":"c01f6b4a-5666-4734-b403-3818692c3a74","Type":"ContainerStarted","Data":"a940694d37c0a2d710cef3d01237dce666e4504220f138a348ba737f5ef632e6"} Oct 30 01:28:36.680150 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:36.680128 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" event={"ID":"c01f6b4a-5666-4734-b403-3818692c3a74","Type":"ContainerStarted","Data":"ad133089c1d32c55951442d42e69fb8efdaaeb235e1766897fca857e3b5fbec0"} Oct 30 01:28:36.680327 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:36.680179 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" event={"ID":"c01f6b4a-5666-4734-b403-3818692c3a74","Type":"ContainerStarted","Data":"f950e1f066644ecfa9333a97331de1137372e1e740e0cbf6435503952ecf6fdf"} Oct 30 01:28:36.680327 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:36.680225 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" event={"ID":"c01f6b4a-5666-4734-b403-3818692c3a74","Type":"ContainerStarted","Data":"da835053494dfe6eac47314a5d42dc4b10dbc06b84306f9448a26407e34e3bef"} Oct 30 01:28:36.753747 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:36.753676 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:36.863335 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:36.863305 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:36.975840 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:36.975808 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:37.076146 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:37.076044 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs\") pod \"network-metrics-daemon-plsqk\" (UID: \"6ff44b73-9f64-4fca-95f8-6a5a24540843\") " pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:37.076146 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:37.076083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:37.076325 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:37.076181 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:37.076325 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:37.076182 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:37.076325 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:37.076231 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs podName:6ff44b73-9f64-4fca-95f8-6a5a24540843 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:53.07621745 +0000 UTC m=+34.213093537 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs") pod "network-metrics-daemon-plsqk" (UID: "6ff44b73-9f64-4fca-95f8-6a5a24540843") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:37.076325 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:37.076248 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret podName:e0bbf87d-7baa-4b99-9b80-695c2c99c4c8 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:53.076241083 +0000 UTC m=+34.213117157 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret") pod "global-pull-secret-syncer-55kfb" (UID: "e0bbf87d-7baa-4b99-9b80-695c2c99c4c8") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:37.176976 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:37.176945 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqrp9\" (UniqueName: \"kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9\") pod \"network-check-target-pct74\" (UID: \"8af570c7-62ff-4f33-93df-1714b5ffc20c\") " pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:37.177186 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:37.177142 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:37.177186 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:37.177165 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:37.177186 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:37.177177 2582 projected.go:196] Error preparing data for projected volume kube-api-access-xqrp9 for pod openshift-network-diagnostics/network-check-target-pct74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:37.177355 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:37.177237 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9 podName:8af570c7-62ff-4f33-93df-1714b5ffc20c nodeName:}" failed. No retries permitted until 2026-10-30 01:28:53.177219045 +0000 UTC m=+34.314095136 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqrp9" (UniqueName: "kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9") pod "network-check-target-pct74" (UID: "8af570c7-62ff-4f33-93df-1714b5ffc20c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:37.467203 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:37.467177 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:37.467372 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:37.467272 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:37.684546 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:37.684515 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:37.792713 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:37.792632 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:37.914785 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:37.914754 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:38.111316 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:38.111293 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:38.224314 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:38.224273 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:38.467221 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:38.467194 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:38.467404 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:38.467194 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:38.467404 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:38.467305 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:38.467404 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:38.467357 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:38.687010 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:38.686980 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" event={"ID":"c01f6b4a-5666-4734-b403-3818692c3a74","Type":"ContainerStarted","Data":"0df45fc375dd1f95986f17f30df9535e7b9bacc094fedace3aac8d33ef3c404a"} Oct 30 01:28:38.751640 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:38.751535 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:39.221343 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:39.221316 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8wd8v_174be2db-9c13-45fe-b91f-c3302ecb57e2/dns-node-resolver/0.log" Oct 30 01:28:39.308206 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:39.308169 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:39.436996 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:39.436947 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:39.467536 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:39.467507 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:39.467977 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:39.467590 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:39.567644 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:39.567548 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:39.692575 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:39.692543 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:40.175301 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:40.175279 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:40.211066 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:40.211051 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lq7lj_4f486e9e-aa32-4a9a-877b-eca8dbba7091/node-ca/0.log" Oct 30 01:28:40.467792 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:40.467755 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:40.467792 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:40.467780 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:40.468504 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:40.467868 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:40.468504 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:40.467988 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:40.694900 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:40.694866 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" event={"ID":"c01f6b4a-5666-4734-b403-3818692c3a74","Type":"ContainerStarted","Data":"26af61463b9f7040e3792ad0788cd5affbac4065eea7216cb3b4a8d30a37a976"} Oct 30 01:28:40.695162 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:40.695145 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:40.710550 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:40.710528 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:40.785241 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:40.785122 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:40.786410 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:40.786371 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" podStartSLOduration=7.670885234 podStartE2EDuration="21.786359243s" totalImagesPullingTime="14.115474009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:40.773457839 +0000 UTC m=+21.910333936" watchObservedRunningTime="2026-10-30 01:28:40.786359243 +0000 UTC m=+21.923235338" Oct 30 01:28:40.786745 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:40.786723 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zrv6d" podStartSLOduration=11.774962072 podStartE2EDuration="21.786718506s" totalImagesPullingTime="10.011756434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:28:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:35.748712035 +0000 UTC m=+16.885588130" watchObservedRunningTime="2026-10-30 01:28:40.786718506 +0000 UTC m=+21.923594602" Oct 30 01:28:40.985898 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:40.985867 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:41.129483 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:41.129401 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:41.241858 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:41.241828 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:41.386970 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:41.386865 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:41.467561 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:41.467528 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:41.467731 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:41.467667 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:41.576349 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:41.576319 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:41.697425 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:41.697398 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:28:41.697820 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:41.697803 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:41.712374 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:41.712353 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:41.879578 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:41.879503 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:42.177261 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:42.177229 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:42.466892 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:42.466822 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:42.466892 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:42.466847 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:42.467074 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:42.466940 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:42.467074 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:42.467051 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:42.699505 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:42.699478 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:28:42.733055 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:42.732955 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:43.024707 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:43.024622 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:43.146660 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:43.146483 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:43.213970 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:43.213942 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-44ftt" Oct 30 01:28:43.214137 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:43.214082 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:28:43.214575 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:43.214561 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-44ftt" Oct 30 01:28:43.386082 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:43.385994 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:43.467027 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:43.467003 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:43.467207 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:43.467113 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:43.527457 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:43.527428 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:43.651273 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:43.651204 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:43.701152 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:43.701127 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:28:44.177040 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:44.177010 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:44.320969 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:44.320914 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:44.466851 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:44.466751 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:44.466851 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:44.466777 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:44.467031 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:44.466864 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:44.467031 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:44.466984 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:44.775480 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:44.775372 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:44.979562 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:44.979534 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:45.178888 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:45.178854 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:45.290188 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:45.290164 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:28:45.290384 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:45.290370 2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:28:45.300143 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:45.300108 2582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" podUID="c01f6b4a-5666-4734-b403-3818692c3a74" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 01:28:45.308371 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:45.308343 2582 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" podUID="c01f6b4a-5666-4734-b403-3818692c3a74" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 01:28:45.466941 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:45.466863 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:45.467123 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:45.466977 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:45.573996 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:45.573965 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:46.159674 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:46.159636 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:46.275010 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:46.274962 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:46.385984 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:46.385951 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:46.466988 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:46.466959 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:46.467172 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:46.467058 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:46.467172 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:46.467115 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:46.467247 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:46.467222 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:46.603715 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:46.603679 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:46.775511 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:46.775417 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:46.977624 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:46.977582 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:47.183395 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:47.183354 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:47.467675 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:47.467591 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:47.467824 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:47.467723 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:47.575451 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:47.575421 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:47.814573 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:47.814487 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:47.927125 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:47.927085 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:48.389429 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:48.389393 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:48.467547 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:48.467521 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:48.467720 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:48.467526 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:48.467720 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:48.467613 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:48.467720 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:48.467705 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:48.813332 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:48.813299 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:48.985641 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:48.985609 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:49.174759 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:49.174727 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:49.467555 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:49.467488 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:49.467892 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:49.467572 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:49.533126 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:49.533071 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:49.638369 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:49.638319 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:49.980109 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:49.980056 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:50.183363 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:50.183331 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:50.467603 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:50.467573 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:50.467999 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:50.467671 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:50.467999 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:50.467726 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:50.467999 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:50.467796 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:51.466870 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:51.466839 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:51.467116 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:51.466937 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:52.467579 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:52.467539 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:52.468183 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:52.467539 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:52.468183 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:52.467680 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:52.468183 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:52.467713 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:53.091297 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:53.091265 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs\") pod \"network-metrics-daemon-plsqk\" (UID: \"6ff44b73-9f64-4fca-95f8-6a5a24540843\") " pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:53.091297 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:53.091308 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:53.091503 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:53.091413 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:53.091503 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:53.091414 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:53.091503 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:53.091469 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret podName:e0bbf87d-7baa-4b99-9b80-695c2c99c4c8 nodeName:}" failed. No retries permitted until 2026-10-30 01:29:25.091456814 +0000 UTC m=+66.228332900 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret") pod "global-pull-secret-syncer-55kfb" (UID: "e0bbf87d-7baa-4b99-9b80-695c2c99c4c8") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:28:53.091503 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:53.091485 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs podName:6ff44b73-9f64-4fca-95f8-6a5a24540843 nodeName:}" failed. No retries permitted until 2026-10-30 01:29:25.09147788 +0000 UTC m=+66.228353967 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs") pod "network-metrics-daemon-plsqk" (UID: "6ff44b73-9f64-4fca-95f8-6a5a24540843") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:28:53.191594 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:53.191565 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqrp9\" (UniqueName: \"kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9\") pod \"network-check-target-pct74\" (UID: \"8af570c7-62ff-4f33-93df-1714b5ffc20c\") " pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:53.191714 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:53.191674 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:28:53.191714 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:53.191687 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:28:53.191714 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:53.191697 2582 projected.go:196] Error preparing data for projected volume kube-api-access-xqrp9 for pod openshift-network-diagnostics/network-check-target-pct74: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:53.191811 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:53.191740 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9 podName:8af570c7-62ff-4f33-93df-1714b5ffc20c nodeName:}" failed. No retries permitted until 2026-10-30 01:29:25.191729281 +0000 UTC m=+66.328605368 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xqrp9" (UniqueName: "kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9") pod "network-check-target-pct74" (UID: "8af570c7-62ff-4f33-93df-1714b5ffc20c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:28:53.467195 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:53.467153 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:53.467341 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:53.467307 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:54.467311 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:54.467274 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:54.467311 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:54.467307 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:54.467745 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:54.467382 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:54.467745 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:54.467437 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:55.466989 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:55.466956 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:55.467174 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:55.467061 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:56.467205 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:56.467170 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:56.467667 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:56.467282 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:56.467667 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:56.467345 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:56.467667 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:56.467458 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:57.467699 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:57.467668 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:57.468112 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:57.467769 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:58.467442 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:58.467410 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:28:58.467608 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:58.467448 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:28:58.467608 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:58.467528 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:28:58.467686 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:58.467624 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:28:59.467654 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:59.467623 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:28:59.468033 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:28:59.467724 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:28:59.588719 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:59.588660 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:59.704881 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:59.704825 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:59.815022 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:59.814942 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:28:59.921974 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:28:59.921931 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:00.032502 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:00.032472 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:00.467321 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:00.467291 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:29:00.467478 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:00.467291 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:29:00.467478 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:29:00.467426 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:29:00.467478 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:29:00.467456 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:29:01.303853 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:01.303818 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-55kfb"] Oct 30 01:29:01.304331 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:01.303945 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:29:01.304331 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:29:01.304036 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:29:01.307197 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:01.307157 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pct74"] Oct 30 01:29:01.307350 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:01.307271 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:29:01.307416 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:29:01.307374 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:29:01.307908 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:01.307885 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-plsqk"] Oct 30 01:29:01.308156 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:01.308135 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:29:01.308259 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:29:01.308244 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:29:01.743477 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:01.743451 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8wd8v_174be2db-9c13-45fe-b91f-c3302ecb57e2/dns-node-resolver/0.log" Oct 30 01:29:02.467686 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:02.467607 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:29:02.468043 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:29:02.467717 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:29:02.701221 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:02.701190 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:02.815370 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:02.815267 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:02.924737 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:02.924701 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:02.943545 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:02.943523 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lq7lj_4f486e9e-aa32-4a9a-877b-eca8dbba7091/node-ca/0.log" Oct 30 01:29:03.093686 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:03.093592 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:03.467186 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:03.467154 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:29:03.467477 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:29:03.467265 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:29:03.467477 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:03.467305 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:29:03.467477 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:29:03.467412 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:29:04.450517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:04.450300 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:04.466955 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:04.466930 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:29:04.467117 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:29:04.467024 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-55kfb" podUID="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" Oct 30 01:29:05.186181 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:05.186133 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:05.461261 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:05.461178 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:05.467376 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:05.467353 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:29:05.467694 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:29:05.467660 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pct74" podUID="8af570c7-62ff-4f33-93df-1714b5ffc20c" Oct 30 01:29:05.467793 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:05.467707 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:29:05.467846 ip-10-0-130-0 kubenswrapper[2582]: E1030 01:29:05.467809 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-plsqk" podUID="6ff44b73-9f64-4fca-95f8-6a5a24540843" Oct 30 01:29:05.565931 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:05.565894 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:06.183913 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.183887 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeReady" Oct 30 01:29:06.184043 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.184005 2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 01:29:06.251634 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.251601 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tlpbm"] Oct 30 01:29:06.253559 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.253543 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tlpbm" Oct 30 01:29:06.258005 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.257978 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vccw5"] Oct 30 01:29:06.259691 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.259672 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:06.272261 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.272237 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6vkc5"] Oct 30 01:29:06.273854 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.273835 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:06.275077 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.275058 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 01:29:06.275242 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.275225 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 01:29:06.275315 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.275232 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zqs4x\"" Oct 30 01:29:06.279041 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.279025 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 01:29:06.280756 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.280732 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tlpbm"] Oct 30 01:29:06.286597 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.286058 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vccw5"] Oct 30 01:29:06.298115 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.298067 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6vkc5"] Oct 30 01:29:06.375537 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.375418 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 01:29:06.375537 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.375419 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 01:29:06.375537 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.375439 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 01:29:06.375749 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.375550 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 01:29:06.375749 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.375561 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2p848\"" Oct 30 01:29:06.375749 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.375603 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 01:29:06.375749 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.375613 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vfcpg\"" Oct 30 01:29:06.375749 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.375606 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 01:29:06.378963 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.378946 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91d108c9-4e8c-4d00-8e42-1eecd52741a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:06.379012 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.378983 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/91d108c9-4e8c-4d00-8e42-1eecd52741a2-crio-socket\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:06.379012 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.379002 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da638295-de7f-462a-9a14-2b8105e636b6-config-volume\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:06.379138 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.379121 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/91d108c9-4e8c-4d00-8e42-1eecd52741a2-data-volume\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:06.379193 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.379157 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91d108c9-4e8c-4d00-8e42-1eecd52741a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:06.379193 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.379184 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hwlz\" (UniqueName: \"kubernetes.io/projected/91d108c9-4e8c-4d00-8e42-1eecd52741a2-kube-api-access-2hwlz\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:06.379264 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.379222 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da638295-de7f-462a-9a14-2b8105e636b6-metrics-tls\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:06.379297 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.379273 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da638295-de7f-462a-9a14-2b8105e636b6-tmp-dir\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:06.379297 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.379290 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scl9k\" (UniqueName: \"kubernetes.io/projected/da638295-de7f-462a-9a14-2b8105e636b6-kube-api-access-scl9k\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:06.379394 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.379308 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bde30efc-28f6-4fb1-83b7-2b59d6d34359-cert\") pod \"ingress-canary-tlpbm\" (UID: \"bde30efc-28f6-4fb1-83b7-2b59d6d34359\") " pod="openshift-ingress-canary/ingress-canary-tlpbm" Oct 30 01:29:06.379444 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.379404 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd6sl\" (UniqueName: \"kubernetes.io/projected/bde30efc-28f6-4fb1-83b7-2b59d6d34359-kube-api-access-wd6sl\") pod \"ingress-canary-tlpbm\" (UID: \"bde30efc-28f6-4fb1-83b7-2b59d6d34359\") " pod="openshift-ingress-canary/ingress-canary-tlpbm" Oct 30 01:29:06.466801 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.466766 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:29:06.480537 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.480515 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/91d108c9-4e8c-4d00-8e42-1eecd52741a2-data-volume\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:06.480658 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.480543 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91d108c9-4e8c-4d00-8e42-1eecd52741a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:06.480658 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.480559 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hwlz\" (UniqueName: \"kubernetes.io/projected/91d108c9-4e8c-4d00-8e42-1eecd52741a2-kube-api-access-2hwlz\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:06.480658 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.480574 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da638295-de7f-462a-9a14-2b8105e636b6-metrics-tls\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:06.480658 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.480593 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da638295-de7f-462a-9a14-2b8105e636b6-tmp-dir\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:06.480658 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.480607 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scl9k\" (UniqueName: \"kubernetes.io/projected/da638295-de7f-462a-9a14-2b8105e636b6-kube-api-access-scl9k\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:06.480658 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.480626 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bde30efc-28f6-4fb1-83b7-2b59d6d34359-cert\") pod \"ingress-canary-tlpbm\" (UID: \"bde30efc-28f6-4fb1-83b7-2b59d6d34359\") " pod="openshift-ingress-canary/ingress-canary-tlpbm" Oct 30 01:29:06.480658 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.480659 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd6sl\" (UniqueName: \"kubernetes.io/projected/bde30efc-28f6-4fb1-83b7-2b59d6d34359-kube-api-access-wd6sl\") pod \"ingress-canary-tlpbm\" (UID: \"bde30efc-28f6-4fb1-83b7-2b59d6d34359\") " pod="openshift-ingress-canary/ingress-canary-tlpbm" Oct 30 01:29:06.481006 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.480711 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91d108c9-4e8c-4d00-8e42-1eecd52741a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:06.481006 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.480744 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/91d108c9-4e8c-4d00-8e42-1eecd52741a2-crio-socket\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:06.481006 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.480765 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da638295-de7f-462a-9a14-2b8105e636b6-config-volume\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:06.481006 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.480907 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/91d108c9-4e8c-4d00-8e42-1eecd52741a2-data-volume\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:06.481006 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.480953 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da638295-de7f-462a-9a14-2b8105e636b6-tmp-dir\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:06.481269 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.481085 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/91d108c9-4e8c-4d00-8e42-1eecd52741a2-crio-socket\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:06.481430 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.481410 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91d108c9-4e8c-4d00-8e42-1eecd52741a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:06.481491 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.481433 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da638295-de7f-462a-9a14-2b8105e636b6-config-volume\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:06.484193 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.484167 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nc846\"" Oct 30 01:29:06.484307 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.484190 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 01:29:06.484422 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.484401 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da638295-de7f-462a-9a14-2b8105e636b6-metrics-tls\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:06.484422 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.484417 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91d108c9-4e8c-4d00-8e42-1eecd52741a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:06.484571 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.484554 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bde30efc-28f6-4fb1-83b7-2b59d6d34359-cert\") pod \"ingress-canary-tlpbm\" (UID: \"bde30efc-28f6-4fb1-83b7-2b59d6d34359\") " pod="openshift-ingress-canary/ingress-canary-tlpbm" Oct 30 01:29:06.505533 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.505503 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scl9k\" (UniqueName: \"kubernetes.io/projected/da638295-de7f-462a-9a14-2b8105e636b6-kube-api-access-scl9k\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:06.505636 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.505540 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hwlz\" (UniqueName: \"kubernetes.io/projected/91d108c9-4e8c-4d00-8e42-1eecd52741a2-kube-api-access-2hwlz\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:06.505636 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.505579 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd6sl\" (UniqueName: \"kubernetes.io/projected/bde30efc-28f6-4fb1-83b7-2b59d6d34359-kube-api-access-wd6sl\") pod \"ingress-canary-tlpbm\" (UID: \"bde30efc-28f6-4fb1-83b7-2b59d6d34359\") " pod="openshift-ingress-canary/ingress-canary-tlpbm" Oct 30 01:29:06.564029 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.563994 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tlpbm" Oct 30 01:29:06.573192 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.573163 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:06.586286 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.586254 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:06.892827 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.892756 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tlpbm"] Oct 30 01:29:06.894975 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.894950 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vccw5"] Oct 30 01:29:06.896835 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:29:06.896811 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde30efc_28f6_4fb1_83b7_2b59d6d34359.slice/crio-be9d7b5fa48523697c0914953d02bd598c0b0a4d40650a9a1edc5964b8372ad7 WatchSource:0}: Error finding container be9d7b5fa48523697c0914953d02bd598c0b0a4d40650a9a1edc5964b8372ad7: Status 404 returned error can't find the container with id be9d7b5fa48523697c0914953d02bd598c0b0a4d40650a9a1edc5964b8372ad7 Oct 30 01:29:06.898749 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:29:06.898724 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d108c9_4e8c_4d00_8e42_1eecd52741a2.slice/crio-ec7aa30a140f804d41a551deb79a6db69982d52daedfe91b2a33a41c0470c447 WatchSource:0}: Error finding container ec7aa30a140f804d41a551deb79a6db69982d52daedfe91b2a33a41c0470c447: Status 404 returned error can't find the container with id ec7aa30a140f804d41a551deb79a6db69982d52daedfe91b2a33a41c0470c447 Oct 30 01:29:06.911917 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:06.911893 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6vkc5"] Oct 30 01:29:06.925032 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:29:06.925006 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda638295_de7f_462a_9a14_2b8105e636b6.slice/crio-edbaaa39931f564c22987e5ef6df31a82673dffe17e97708e4661be6d19b523b WatchSource:0}: Error finding container edbaaa39931f564c22987e5ef6df31a82673dffe17e97708e4661be6d19b523b: Status 404 returned error can't find the container with id edbaaa39931f564c22987e5ef6df31a82673dffe17e97708e4661be6d19b523b Oct 30 01:29:07.172175 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.172143 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:07.277878 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.277838 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:07.393241 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.393207 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:07.442022 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.441943 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq"] Oct 30 01:29:07.443988 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.443962 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:07.467326 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.467254 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:29:07.468331 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.467634 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:29:07.469226 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.468879 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 01:29:07.469226 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.468956 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 01:29:07.469226 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.469060 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 01:29:07.469226 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.469132 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 01:29:07.478338 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.478299 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq"] Oct 30 01:29:07.483592 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.483570 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 01:29:07.483754 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.483718 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-lzvbc\"" Oct 30 01:29:07.495897 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.495364 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nd754"] Oct 30 01:29:07.499181 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.499155 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.513233 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.513008 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 01:29:07.513233 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.513008 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hp2kr\"" Oct 30 01:29:07.513233 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.513183 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sfxzn\"" Oct 30 01:29:07.520680 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.520648 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 01:29:07.520822 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.520729 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 01:29:07.520822 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.520736 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tntm7\"" Oct 30 01:29:07.531830 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.531772 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:07.567963 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.567931 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 01:29:07.568219 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.568177 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 01:29:07.568397 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.568380 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 01:29:07.591068 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.591026 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4c33058-34c5-49f3-b803-984c54ff5dad-metrics-client-ca\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.591234 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.591072 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.591294 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.591258 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-textfile\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.591353 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.591300 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43e8ec8c-149e-4257-a891-7b117a79c319-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:07.591407 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.591374 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43e8ec8c-149e-4257-a891-7b117a79c319-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:07.591507 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.591483 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4c33058-34c5-49f3-b803-984c54ff5dad-sys\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.591583 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.591517 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b4c33058-34c5-49f3-b803-984c54ff5dad-root\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.591644 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.591586 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-wtmp\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.591644 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.591628 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssfgz\" (UniqueName: \"kubernetes.io/projected/43e8ec8c-149e-4257-a891-7b117a79c319-kube-api-access-ssfgz\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:07.591789 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.591758 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-tls\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.591843 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.591798 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-accelerators-collector-config\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.591843 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.591823 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqpgm\" (UniqueName: \"kubernetes.io/projected/b4c33058-34c5-49f3-b803-984c54ff5dad-kube-api-access-zqpgm\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.591942 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.591845 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43e8ec8c-149e-4257-a891-7b117a79c319-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:07.646983 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.646952 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:07.692761 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.692700 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-tls\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.692878 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.692768 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-accelerators-collector-config\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.692878 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.692795 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqpgm\" (UniqueName: \"kubernetes.io/projected/b4c33058-34c5-49f3-b803-984c54ff5dad-kube-api-access-zqpgm\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.692878 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.692819 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43e8ec8c-149e-4257-a891-7b117a79c319-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:07.692878 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.692861 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4c33058-34c5-49f3-b803-984c54ff5dad-metrics-client-ca\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.693081 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.692885 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.693081 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.692916 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-textfile\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.693081 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.692943 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43e8ec8c-149e-4257-a891-7b117a79c319-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:07.693081 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.692969 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43e8ec8c-149e-4257-a891-7b117a79c319-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:07.693081 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.692999 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4c33058-34c5-49f3-b803-984c54ff5dad-sys\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.693081 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.693024 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b4c33058-34c5-49f3-b803-984c54ff5dad-root\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.693081 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.693053 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-wtmp\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.693432 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.693103 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssfgz\" (UniqueName: \"kubernetes.io/projected/43e8ec8c-149e-4257-a891-7b117a79c319-kube-api-access-ssfgz\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:07.693840 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.693812 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-textfile\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.694108 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.694067 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b4c33058-34c5-49f3-b803-984c54ff5dad-root\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.694206 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.694186 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4c33058-34c5-49f3-b803-984c54ff5dad-sys\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.694397 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.694366 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4c33058-34c5-49f3-b803-984c54ff5dad-metrics-client-ca\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.694541 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.694515 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-wtmp\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.694603 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.694592 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-accelerators-collector-config\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.695231 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.695205 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43e8ec8c-149e-4257-a891-7b117a79c319-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:07.697166 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.697142 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.697783 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.697756 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43e8ec8c-149e-4257-a891-7b117a79c319-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:07.698741 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.698714 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-tls\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.699135 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.699113 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43e8ec8c-149e-4257-a891-7b117a79c319-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:07.721425 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.721398 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssfgz\" (UniqueName: \"kubernetes.io/projected/43e8ec8c-149e-4257-a891-7b117a79c319-kube-api-access-ssfgz\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:07.722001 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.721980 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqpgm\" (UniqueName: \"kubernetes.io/projected/b4c33058-34c5-49f3-b803-984c54ff5dad-kube-api-access-zqpgm\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.751828 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.751795 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vccw5" event={"ID":"91d108c9-4e8c-4d00-8e42-1eecd52741a2","Type":"ContainerStarted","Data":"a37a93b5c1c5853bb03cfc7d24e2d6a38f14ee14c90b772f7a2d8f50fdd09c87"} Oct 30 01:29:07.751953 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.751863 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vccw5" event={"ID":"91d108c9-4e8c-4d00-8e42-1eecd52741a2","Type":"ContainerStarted","Data":"deec66f9085101f20658c7f26eed98589e01ce8f3f967353f2b09843efd18303"} Oct 30 01:29:07.751953 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.751892 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vccw5" event={"ID":"91d108c9-4e8c-4d00-8e42-1eecd52741a2","Type":"ContainerStarted","Data":"ec7aa30a140f804d41a551deb79a6db69982d52daedfe91b2a33a41c0470c447"} Oct 30 01:29:07.754384 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.754361 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tlpbm" event={"ID":"bde30efc-28f6-4fb1-83b7-2b59d6d34359","Type":"ContainerStarted","Data":"be9d7b5fa48523697c0914953d02bd598c0b0a4d40650a9a1edc5964b8372ad7"} Oct 30 01:29:07.755493 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.755468 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6vkc5" event={"ID":"da638295-de7f-462a-9a14-2b8105e636b6","Type":"ContainerStarted","Data":"edbaaa39931f564c22987e5ef6df31a82673dffe17e97708e4661be6d19b523b"} Oct 30 01:29:07.761437 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.761413 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:07.763014 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.762967 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:07.825100 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.823276 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:07.836864 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:29:07.836834 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c33058_34c5_49f3_b803_984c54ff5dad.slice/crio-30185886dd6355130b98e7f2717a94945ff17d24f2d204ed8bf3d26c2fd6cfcf WatchSource:0}: Error finding container 30185886dd6355130b98e7f2717a94945ff17d24f2d204ed8bf3d26c2fd6cfcf: Status 404 returned error can't find the container with id 30185886dd6355130b98e7f2717a94945ff17d24f2d204ed8bf3d26c2fd6cfcf Oct 30 01:29:07.868464 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.868417 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:07.996813 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:07.996684 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:08.086209 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:08.086167 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq"] Oct 30 01:29:08.666912 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:08.666879 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:08.760761 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:08.760651 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nd754" event={"ID":"b4c33058-34c5-49f3-b803-984c54ff5dad","Type":"ContainerStarted","Data":"30185886dd6355130b98e7f2717a94945ff17d24f2d204ed8bf3d26c2fd6cfcf"} Oct 30 01:29:08.763146 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:08.763110 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" event={"ID":"43e8ec8c-149e-4257-a891-7b117a79c319","Type":"ContainerStarted","Data":"cc4d829c860cc00580d62d90836272b607d7113c1d04f409cb5ad87d6ab83185"} Oct 30 01:29:08.763283 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:08.763173 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" event={"ID":"43e8ec8c-149e-4257-a891-7b117a79c319","Type":"ContainerStarted","Data":"194e4de55221818727cc9de62bf579a04f6ad8a098ba31a251600ef97dcb0c5c"} Oct 30 01:29:08.763283 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:08.763201 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" event={"ID":"43e8ec8c-149e-4257-a891-7b117a79c319","Type":"ContainerStarted","Data":"27371582628e07d070fe27c81ce4db4d80439cbe02c18b552ebb7dffab3d915b"} Oct 30 01:29:08.789432 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:08.789396 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:08.945976 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:08.945940 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:09.285307 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.285210 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:09.488176 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.488142 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:09.769933 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.769843 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" event={"ID":"43e8ec8c-149e-4257-a891-7b117a79c319","Type":"ContainerStarted","Data":"5ca20c00cf8e80933c0883441eaa478a86fa556c4d941f86a381522bccb9802a"} Oct 30 01:29:09.772750 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.772695 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vccw5" event={"ID":"91d108c9-4e8c-4d00-8e42-1eecd52741a2","Type":"ContainerStarted","Data":"874210dc7ae791e11d7baa7dfcab526ac73724409817c361b30412c59cb363be"} Oct 30 01:29:09.775407 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.774822 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6vkc5" event={"ID":"da638295-de7f-462a-9a14-2b8105e636b6","Type":"ContainerStarted","Data":"88fe8bbc9d13dab3e5cc7c3fb969e00bd77241258ffd43add759a63e9894b1ba"} Oct 30 01:29:09.775407 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.774888 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6vkc5" event={"ID":"da638295-de7f-462a-9a14-2b8105e636b6","Type":"ContainerStarted","Data":"59e181b772d38fd4082c937bd2a6b0aec7937a8093aa1a55c2d398820999e31c"} Oct 30 01:29:09.775407 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.774917 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:09.776351 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.776332 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-767946cb5-q64ns"] Oct 30 01:29:09.777245 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.777201 2582 generic.go:356] "Generic (PLEG): container finished" podID="b4c33058-34c5-49f3-b803-984c54ff5dad" containerID="e3d9205c4b9f3b7afa6e7dd89b8290b46407ad48099b5fe14fca6c98e44805be" exitCode=0 Oct 30 01:29:09.779855 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.779822 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nd754" event={"ID":"b4c33058-34c5-49f3-b803-984c54ff5dad","Type":"ContainerDied","Data":"e3d9205c4b9f3b7afa6e7dd89b8290b46407ad48099b5fe14fca6c98e44805be"} Oct 30 01:29:09.780018 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.779943 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:09.797174 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.797150 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 01:29:09.807911 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.807875 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-767946cb5-q64ns"] Oct 30 01:29:09.825355 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.825308 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" podStartSLOduration=1.551224122 podStartE2EDuration="2.825295536s" totalImagesPullingTime="1.274071414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:09.824252758 +0000 UTC m=+50.961128855" watchObservedRunningTime="2026-10-30 01:29:09.825295536 +0000 UTC m=+50.962171629" Oct 30 01:29:09.873980 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.873929 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vccw5" podStartSLOduration=1.515929743 podStartE2EDuration="3.873914486s" totalImagesPullingTime="2.357984743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:29:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:09.87289895 +0000 UTC m=+51.009775070" watchObservedRunningTime="2026-10-30 01:29:09.873914486 +0000 UTC m=+51.010790576" Oct 30 01:29:09.897669 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.897575 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 01:29:09.897669 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.897596 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 01:29:09.897669 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.897631 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 01:29:09.897669 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.897629 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 01:29:09.897839 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.897678 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xk9xs\"" Oct 30 01:29:09.897839 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.897747 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3a819liq0v26f\"" Oct 30 01:29:09.913772 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.913745 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:09.913902 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.913778 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-grpc-tls\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:09.913955 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.913908 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fa1c18e-5933-4e09-85f8-28033e6cc4df-metrics-client-ca\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:09.914020 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.913999 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:09.914306 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.914285 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n55vv\" (UniqueName: \"kubernetes.io/projected/4fa1c18e-5933-4e09-85f8-28033e6cc4df-kube-api-access-n55vv\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:09.914405 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.914324 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-tls\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:09.914405 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.914353 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:09.914508 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.914467 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:09.915066 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:09.914942 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fa1c18e-5933-4e09-85f8-28033e6cc4df-tmp\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.016280 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.016253 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fa1c18e-5933-4e09-85f8-28033e6cc4df-tmp\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.016386 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.016301 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.016386 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.016331 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-grpc-tls\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.016386 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.016365 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fa1c18e-5933-4e09-85f8-28033e6cc4df-metrics-client-ca\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.016548 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.016397 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.016548 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.016434 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n55vv\" (UniqueName: \"kubernetes.io/projected/4fa1c18e-5933-4e09-85f8-28033e6cc4df-kube-api-access-n55vv\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.016548 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.016459 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-tls\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.016548 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.016484 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.016548 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.016508 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.016777 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.016609 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fa1c18e-5933-4e09-85f8-28033e6cc4df-tmp\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.017915 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.017886 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fa1c18e-5933-4e09-85f8-28033e6cc4df-metrics-client-ca\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.019150 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.019126 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.019254 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.019130 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.019536 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.019517 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-grpc-tls\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.019618 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.019593 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.019667 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.019643 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-tls\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.019737 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.019706 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.025724 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.025674 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-6vkc5" podStartSLOduration=2.094914329 podStartE2EDuration="4.025658826s" totalImagesPullingTime="1.930744497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:10.02499591 +0000 UTC m=+51.161872006" watchObservedRunningTime="2026-10-30 01:29:10.025658826 +0000 UTC m=+51.162534924" Oct 30 01:29:10.042116 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.042077 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n55vv\" (UniqueName: \"kubernetes.io/projected/4fa1c18e-5933-4e09-85f8-28033e6cc4df-kube-api-access-n55vv\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.107257 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.107217 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:10.425668 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.425632 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-767946cb5-q64ns"] Oct 30 01:29:10.428956 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:29:10.428930 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa1c18e_5933_4e09_85f8_28033e6cc4df.slice/crio-ec810a6918efaf59b46323d1b76630d9d0495662bed0d79cd8f83a6a10330e34 WatchSource:0}: Error finding container ec810a6918efaf59b46323d1b76630d9d0495662bed0d79cd8f83a6a10330e34: Status 404 returned error can't find the container with id ec810a6918efaf59b46323d1b76630d9d0495662bed0d79cd8f83a6a10330e34 Oct 30 01:29:10.752810 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.752738 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:10.781784 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.781755 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" event={"ID":"4fa1c18e-5933-4e09-85f8-28033e6cc4df","Type":"ContainerStarted","Data":"ec810a6918efaf59b46323d1b76630d9d0495662bed0d79cd8f83a6a10330e34"} Oct 30 01:29:10.783725 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.783700 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nd754" event={"ID":"b4c33058-34c5-49f3-b803-984c54ff5dad","Type":"ContainerStarted","Data":"05de9d409dd41d5111f9d2117a7b812f3a701b6c82deb678aad389bc714f3ec7"} Oct 30 01:29:10.783864 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.783750 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nd754" event={"ID":"b4c33058-34c5-49f3-b803-984c54ff5dad","Type":"ContainerStarted","Data":"88a18b66beb5032b72ce1505fdb9a1d938881dc108df1c5b2824003413b9d89f"} Oct 30 01:29:10.785278 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.785253 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tlpbm" event={"ID":"bde30efc-28f6-4fb1-83b7-2b59d6d34359","Type":"ContainerStarted","Data":"0f6db6c8c61f6b585c90690b3755784e81aa4f8540260ca3d09fcd599c2fa12c"} Oct 30 01:29:10.833315 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.833255 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nd754" podStartSLOduration=2.55652252 podStartE2EDuration="3.833239823s" totalImagesPullingTime="1.18224877s" totalInitContainerRuntime="94.468533ms" isStatefulPod=true podCreationTimestamp="2026-10-30 01:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:10.832125888 +0000 UTC m=+51.969001974" watchObservedRunningTime="2026-10-30 01:29:10.833239823 +0000 UTC m=+51.970115918" Oct 30 01:29:10.858263 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.858226 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:10.878750 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.878698 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tlpbm" podStartSLOduration=1.802795344 podStartE2EDuration="4.878683753s" totalImagesPullingTime="3.075888409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:10.877574885 +0000 UTC m=+52.014450984" watchObservedRunningTime="2026-10-30 01:29:10.878683753 +0000 UTC m=+52.015559848" Oct 30 01:29:10.969917 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:10.969874 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:11.976956 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:11.976923 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-868b7dc86d-hzrmz"] Oct 30 01:29:11.979014 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:11.978998 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:11.998930 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:11.998908 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rgh9c\"" Oct 30 01:29:12.013510 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.013486 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-435e5nstmdbes\"" Oct 30 01:29:12.013626 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.013551 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 01:29:12.016360 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.016338 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 01:29:12.017396 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.017377 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-868b7dc86d-hzrmz"] Oct 30 01:29:12.099170 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.099135 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 01:29:12.099325 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.099279 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 01:29:12.132673 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.132644 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3657b83a-1cdf-44de-9a9d-2800cfae5524-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.132806 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.132678 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtbks\" (UniqueName: \"kubernetes.io/projected/3657b83a-1cdf-44de-9a9d-2800cfae5524-kube-api-access-xtbks\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.132806 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.132704 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3657b83a-1cdf-44de-9a9d-2800cfae5524-secret-metrics-server-client-certs\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.132889 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.132858 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3657b83a-1cdf-44de-9a9d-2800cfae5524-client-ca-bundle\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.132939 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.132910 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3657b83a-1cdf-44de-9a9d-2800cfae5524-audit-log\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.132991 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.132949 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3657b83a-1cdf-44de-9a9d-2800cfae5524-secret-metrics-server-tls\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.132991 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.132976 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3657b83a-1cdf-44de-9a9d-2800cfae5524-metrics-server-audit-profiles\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.233960 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.233873 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3657b83a-1cdf-44de-9a9d-2800cfae5524-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.233960 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.233908 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtbks\" (UniqueName: \"kubernetes.io/projected/3657b83a-1cdf-44de-9a9d-2800cfae5524-kube-api-access-xtbks\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.233960 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.233929 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3657b83a-1cdf-44de-9a9d-2800cfae5524-secret-metrics-server-client-certs\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.234245 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.233963 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3657b83a-1cdf-44de-9a9d-2800cfae5524-client-ca-bundle\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.234245 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.234000 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3657b83a-1cdf-44de-9a9d-2800cfae5524-audit-log\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.234245 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.234048 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3657b83a-1cdf-44de-9a9d-2800cfae5524-secret-metrics-server-tls\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.234245 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.234072 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3657b83a-1cdf-44de-9a9d-2800cfae5524-metrics-server-audit-profiles\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.235244 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.234586 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3657b83a-1cdf-44de-9a9d-2800cfae5524-audit-log\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.235244 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.235176 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3657b83a-1cdf-44de-9a9d-2800cfae5524-metrics-server-audit-profiles\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.235244 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.235189 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3657b83a-1cdf-44de-9a9d-2800cfae5524-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.236745 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.236723 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3657b83a-1cdf-44de-9a9d-2800cfae5524-client-ca-bundle\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.237129 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.237108 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3657b83a-1cdf-44de-9a9d-2800cfae5524-secret-metrics-server-client-certs\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.237328 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.237305 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3657b83a-1cdf-44de-9a9d-2800cfae5524-secret-metrics-server-tls\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.259405 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.259380 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtbks\" (UniqueName: \"kubernetes.io/projected/3657b83a-1cdf-44de-9a9d-2800cfae5524-kube-api-access-xtbks\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.289279 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.289251 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:12.297047 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.296942 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:12.440000 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.439956 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:12.632242 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.632162 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-868b7dc86d-hzrmz"] Oct 30 01:29:12.636215 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:29:12.636186 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3657b83a_1cdf_44de_9a9d_2800cfae5524.slice/crio-9edf2b9801239a3c71c8e460fc7a1de9fc4ceefb1af1d5affc9bcd50e7423876 WatchSource:0}: Error finding container 9edf2b9801239a3c71c8e460fc7a1de9fc4ceefb1af1d5affc9bcd50e7423876: Status 404 returned error can't find the container with id 9edf2b9801239a3c71c8e460fc7a1de9fc4ceefb1af1d5affc9bcd50e7423876 Oct 30 01:29:12.793038 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:12.793004 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" event={"ID":"3657b83a-1cdf-44de-9a9d-2800cfae5524","Type":"ContainerStarted","Data":"9edf2b9801239a3c71c8e460fc7a1de9fc4ceefb1af1d5affc9bcd50e7423876"} Oct 30 01:29:13.799482 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:13.799451 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" event={"ID":"4fa1c18e-5933-4e09-85f8-28033e6cc4df","Type":"ContainerStarted","Data":"a09a4fa360972e215f8d5f0c9ebda4f0552c72afc50c40f68ea5f543abe44a46"} Oct 30 01:29:13.800060 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:13.799518 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" event={"ID":"4fa1c18e-5933-4e09-85f8-28033e6cc4df","Type":"ContainerStarted","Data":"50d4f2acd7a89d790aada472ccdb0a344facd579e3cf408438805f0eb97566e8"} Oct 30 01:29:13.800060 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:13.799560 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" event={"ID":"4fa1c18e-5933-4e09-85f8-28033e6cc4df","Type":"ContainerStarted","Data":"b1d53a60a6542e2df386526cbdb7ed965cbbc421573d095e37a7fe1e07bb6d93"} Oct 30 01:29:14.164672 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.164634 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:14.392582 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.392550 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:14.546035 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.545963 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:14.554370 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.554343 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 01:29:14.557026 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.557008 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.579812 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.579775 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 01:29:14.592658 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.592626 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 01:29:14.592927 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.592626 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 01:29:14.592927 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.592635 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 01:29:14.592927 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.592669 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 01:29:14.592927 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.592785 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nw28w\"" Oct 30 01:29:14.592927 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.592832 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 01:29:14.592927 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.592858 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 01:29:14.592927 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.592897 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 01:29:14.592927 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.592912 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 01:29:14.593323 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.592978 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-569c19rarmjmk\"" Oct 30 01:29:14.593323 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.593030 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 01:29:14.594018 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.594001 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 01:29:14.594102 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.594014 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 01:29:14.594102 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.594027 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 01:29:14.622045 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.622019 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 01:29:14.622661 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.622645 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 01:29:14.657270 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657239 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657270 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657274 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657448 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657295 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657448 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657318 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657448 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657436 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657540 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657457 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657540 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657472 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657540 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657493 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657629 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657569 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657629 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657590 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wspxg\" (UniqueName: \"kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-kube-api-access-wspxg\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657629 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657611 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657713 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657628 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657713 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657654 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657713 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657679 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657798 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657721 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657798 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657745 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657852 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657822 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657852 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657841 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657911 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657860 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.657911 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.657875 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.758841 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.758809 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.758841 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.758839 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.759109 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.758858 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.759109 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.758876 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.759109 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.758898 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.759109 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.758922 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.759109 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.759072 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.759359 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.759133 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.759359 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.759159 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.759359 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.759177 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.759359 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.759210 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.760155 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.759821 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.760691 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.760362 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.760965 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.760940 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.761054 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.760994 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.761054 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.761030 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.761185 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.761123 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.761185 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.761153 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.761185 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.761175 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.761334 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.761201 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.761334 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.761258 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.761334 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.761300 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wspxg\" (UniqueName: \"kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-kube-api-access-wspxg\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.761669 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.761641 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.761899 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.761878 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.762281 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.762258 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.762723 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.762694 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.762809 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.762734 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.763160 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.763110 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.763298 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.763275 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.763418 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.763393 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.763575 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.763553 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.763971 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.763927 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.764408 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.764382 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.764513 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.764417 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.764655 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.764630 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.764962 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.764943 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.765456 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.765434 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.765608 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.765591 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.766148 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.766133 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.787556 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.787534 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wspxg\" (UniqueName: \"kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-kube-api-access-wspxg\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.806327 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.806255 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" event={"ID":"4fa1c18e-5933-4e09-85f8-28033e6cc4df","Type":"ContainerStarted","Data":"90516b472a57b809d3bcb56ebd2ce4f1ff4e5b3105a04fdeb99f3ae45c77c3f4"} Oct 30 01:29:14.806663 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.806328 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" event={"ID":"4fa1c18e-5933-4e09-85f8-28033e6cc4df","Type":"ContainerStarted","Data":"27dcb3d7a0fa90d12dc770f60e64ce88a4d6b46f51faf951cb7a61ec5895e638"} Oct 30 01:29:14.806663 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.806362 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" event={"ID":"4fa1c18e-5933-4e09-85f8-28033e6cc4df","Type":"ContainerStarted","Data":"fdb2c1b33b737594295cc87e38918216b9fff0f424d2908de24b6e50de0a735f"} Oct 30 01:29:14.806663 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.806402 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:14.807734 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.807718 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" event={"ID":"3657b83a-1cdf-44de-9a9d-2800cfae5524","Type":"ContainerStarted","Data":"7c5ae9a458e0ecc5ab5ed7b498740e2f67bfc272d20a6303c7176badd3cca1d1"} Oct 30 01:29:14.868509 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.868462 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" podStartSLOduration=2.605022362 podStartE2EDuration="5.868448024s" totalImagesPullingTime="3.263425662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:14.867919369 +0000 UTC m=+56.004795463" watchObservedRunningTime="2026-10-30 01:29:14.868448024 +0000 UTC m=+56.005324120" Oct 30 01:29:14.874870 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.874842 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:14.928753 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:14.928691 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" podStartSLOduration=2.692235065 podStartE2EDuration="3.928669684s" totalImagesPullingTime="1.236434619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:14.927567096 +0000 UTC m=+56.064443195" watchObservedRunningTime="2026-10-30 01:29:14.928669684 +0000 UTC m=+56.065545780" Oct 30 01:29:15.208906 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:15.208755 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 01:29:15.212609 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:29:15.212579 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e3a6b92_65a9_411e_bbf0_bf12b5c9c047.slice/crio-de46b4224869ff221f36081f4ff9119a601df3ef83012041978fcab90cafee19 WatchSource:0}: Error finding container de46b4224869ff221f36081f4ff9119a601df3ef83012041978fcab90cafee19: Status 404 returned error can't find the container with id de46b4224869ff221f36081f4ff9119a601df3ef83012041978fcab90cafee19 Oct 30 01:29:15.309919 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:15.309892 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:15.631650 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:15.631575 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:15.762403 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:15.762361 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:15.813282 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:15.813238 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047","Type":"ContainerStarted","Data":"de46b4224869ff221f36081f4ff9119a601df3ef83012041978fcab90cafee19"} Oct 30 01:29:16.817715 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:16.817684 2582 generic.go:356] "Generic (PLEG): container finished" podID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerID="a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e" exitCode=0 Oct 30 01:29:16.818131 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:16.817734 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047","Type":"ContainerDied","Data":"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e"} Oct 30 01:29:17.800933 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:17.800874 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:17.921679 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:17.921630 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:19.787892 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:19.787864 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:19.834116 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:19.834069 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047","Type":"ContainerStarted","Data":"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe"} Oct 30 01:29:19.834443 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:19.834416 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047","Type":"ContainerStarted","Data":"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac"} Oct 30 01:29:19.834619 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:19.834601 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047","Type":"ContainerStarted","Data":"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754"} Oct 30 01:29:20.248106 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:20.248049 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:20.364964 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:20.364916 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:20.683797 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:20.683766 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:20.820521 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:20.820484 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:20.826439 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:20.826416 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:20.842438 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:20.842350 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047","Type":"ContainerStarted","Data":"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e"} Oct 30 01:29:20.842438 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:20.842434 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047","Type":"ContainerStarted","Data":"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe"} Oct 30 01:29:20.842639 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:20.842476 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047","Type":"ContainerStarted","Data":"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e"} Oct 30 01:29:20.957702 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:20.957630 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:20.998828 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:20.998776 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.246947267 podStartE2EDuration="6.998761605s" totalImagesPullingTime="3.737223397s" totalInitContainerRuntime="14.590941ms" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:20.996992447 +0000 UTC m=+62.133868533" watchObservedRunningTime="2026-10-30 01:29:20.998761605 +0000 UTC m=+62.135637700" Oct 30 01:29:21.073001 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:21.072949 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:21.208859 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:21.208774 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:22.075199 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:22.075166 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:22.276238 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:22.276210 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:22.387921 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:22.387843 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:22.653253 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:22.653153 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:22.760733 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:22.760700 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:24.117609 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:24.117574 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:24.135777 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:24.135748 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:24.857651 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:24.857623 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:24.872383 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:24.872360 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:25.162054 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.162026 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs\") pod \"network-metrics-daemon-plsqk\" (UID: \"6ff44b73-9f64-4fca-95f8-6a5a24540843\") " pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:29:25.162507 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.162070 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:29:25.178297 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.178271 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 01:29:25.178297 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.178290 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 01:29:25.184976 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.184951 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs\") pod \"network-metrics-daemon-plsqk\" (UID: \"6ff44b73-9f64-4fca-95f8-6a5a24540843\") " pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:29:25.185077 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.184989 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:29:25.234930 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.234905 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sfxzn\"" Oct 30 01:29:25.239426 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.239410 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:29:25.262521 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.262487 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqrp9\" (UniqueName: \"kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9\") pod \"network-check-target-pct74\" (UID: \"8af570c7-62ff-4f33-93df-1714b5ffc20c\") " pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:29:25.278517 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.278489 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 01:29:25.299661 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.299628 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 01:29:25.305941 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.305917 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqrp9\" (UniqueName: \"kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9\") pod \"network-check-target-pct74\" (UID: \"8af570c7-62ff-4f33-93df-1714b5ffc20c\") " pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:29:25.392276 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.392247 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nc846\"" Oct 30 01:29:25.397261 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.397241 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:29:25.519982 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.519956 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hp2kr\"" Oct 30 01:29:25.525020 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.525003 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:29:25.535285 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.535260 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-plsqk"] Oct 30 01:29:25.538151 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:29:25.538124 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff44b73_9f64_4fca_95f8_6a5a24540843.slice/crio-cfa1331afcebcca333f0e9ed66f18f2c3d5f10166676c32c7f6642aa5949d5c5 WatchSource:0}: Error finding container cfa1331afcebcca333f0e9ed66f18f2c3d5f10166676c32c7f6642aa5949d5c5: Status 404 returned error can't find the container with id cfa1331afcebcca333f0e9ed66f18f2c3d5f10166676c32c7f6642aa5949d5c5 Oct 30 01:29:25.690884 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.690847 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-55kfb"] Oct 30 01:29:25.693543 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:29:25.693481 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0bbf87d_7baa_4b99_9b80_695c2c99c4c8.slice/crio-637002c490455d177e3c95d7933023979dc9946e4ffe05bb08f8e03c5d0b92d7 WatchSource:0}: Error finding container 637002c490455d177e3c95d7933023979dc9946e4ffe05bb08f8e03c5d0b92d7: Status 404 returned error can't find the container with id 637002c490455d177e3c95d7933023979dc9946e4ffe05bb08f8e03c5d0b92d7 Oct 30 01:29:25.822284 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.822257 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pct74"] Oct 30 01:29:25.825359 ip-10-0-130-0 kubenswrapper[2582]: W1030 01:29:25.825336 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af570c7_62ff_4f33_93df_1714b5ffc20c.slice/crio-fe7a5c1e60ceb682069567715a19487b2a4e4e34db069de628afd3114e733839 WatchSource:0}: Error finding container fe7a5c1e60ceb682069567715a19487b2a4e4e34db069de628afd3114e733839: Status 404 returned error can't find the container with id fe7a5c1e60ceb682069567715a19487b2a4e4e34db069de628afd3114e733839 Oct 30 01:29:25.861145 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.861115 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pct74" event={"ID":"8af570c7-62ff-4f33-93df-1714b5ffc20c","Type":"ContainerStarted","Data":"fe7a5c1e60ceb682069567715a19487b2a4e4e34db069de628afd3114e733839"} Oct 30 01:29:25.862201 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.862173 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-55kfb" event={"ID":"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8","Type":"ContainerStarted","Data":"637002c490455d177e3c95d7933023979dc9946e4ffe05bb08f8e03c5d0b92d7"} Oct 30 01:29:25.863173 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:25.863150 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-plsqk" event={"ID":"6ff44b73-9f64-4fca-95f8-6a5a24540843","Type":"ContainerStarted","Data":"cfa1331afcebcca333f0e9ed66f18f2c3d5f10166676c32c7f6642aa5949d5c5"} Oct 30 01:29:26.871245 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:26.871155 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-plsqk" event={"ID":"6ff44b73-9f64-4fca-95f8-6a5a24540843","Type":"ContainerStarted","Data":"23b8539d7fc1a8d66edefc8feb9f2ce19143c0cb6fcb5cffc0743d00a03bc158"} Oct 30 01:29:26.871245 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:26.871213 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-plsqk" event={"ID":"6ff44b73-9f64-4fca-95f8-6a5a24540843","Type":"ContainerStarted","Data":"1834c2a279c830c121eab5f55bd0df4b7ce1f21a7161f6be72d946d84e46dc73"} Oct 30 01:29:26.918470 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:26.916930 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-plsqk" podStartSLOduration=66.979755605 podStartE2EDuration="1m7.916912677s" totalImagesPullingTime="937.157072ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:26.916285617 +0000 UTC m=+68.053161714" watchObservedRunningTime="2026-10-30 01:29:26.916912677 +0000 UTC m=+68.053788772" Oct 30 01:29:27.038432 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:27.038315 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:27.205283 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:27.205231 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:27.453699 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:27.453655 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:27.593489 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:27.593408 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:27.827354 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:27.827279 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:27.957675 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:27.957641 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:28.880974 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:28.880940 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pct74" event={"ID":"8af570c7-62ff-4f33-93df-1714b5ffc20c","Type":"ContainerStarted","Data":"b00e9fcbb1572b9758457238b4e24ec22ebd7e42b66ca8f134e1202cd76d0a9a"} Oct 30 01:29:29.885383 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:29.885356 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:29:29.931664 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:29.931613 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pct74" podStartSLOduration=67.959706256 podStartE2EDuration="1m10.9315987s" totalImagesPullingTime="2.971892444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:29.929844009 +0000 UTC m=+71.066720128" watchObservedRunningTime="2026-10-30 01:29:29.9315987 +0000 UTC m=+71.068474796" Oct 30 01:29:30.361794 ip-10-0-130-0 kubenswrapper[2582]: I1030 01:29:30.361724 2582 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:29:30.361890 ip-10-0-130-0 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 01:29:30.370542 ip-10-0-130-0 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 01:29:30.370713 ip-10-0-130-0 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 01:29:30.370762 ip-10-0-130-0 systemd[1]: kubelet.service: Consumed 4.251s CPU time. Oct 30 01:29:30.383586 ip-10-0-130-0 systemd[1]: Starting Kubernetes Kubelet... Oct 30 01:29:30.451822 ip-10-0-130-0 kubenswrapper[5998]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:29:30.451822 ip-10-0-130-0 kubenswrapper[5998]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 01:29:30.451822 ip-10-0-130-0 kubenswrapper[5998]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:29:30.451822 ip-10-0-130-0 kubenswrapper[5998]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:29:30.453991 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.453906 5998 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:29:30.453991 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.453992 5998 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.453997 5998 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454001 5998 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454004 5998 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454007 5998 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454010 5998 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454013 5998 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454016 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454019 5998 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454022 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454025 5998 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454028 5998 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454032 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454034 5998 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454037 5998 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454040 5998 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454043 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454046 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454049 5998 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:29:30.454065 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454057 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454060 5998 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454063 5998 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454066 5998 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454069 5998 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454072 5998 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454075 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454078 5998 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454081 5998 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454083 5998 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454101 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454105 5998 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454110 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454114 5998 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454117 5998 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454120 5998 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454128 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454131 5998 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454134 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454137 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:29:30.454583 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454141 5998 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454143 5998 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454146 5998 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454149 5998 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454151 5998 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454154 5998 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454157 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454160 5998 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454162 5998 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454165 5998 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454168 5998 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454171 5998 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454174 5998 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454176 5998 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454179 5998 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454182 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454185 5998 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454187 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454190 5998 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454193 5998 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:29:30.455057 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454195 5998 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454198 5998 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454201 5998 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454203 5998 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454207 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454211 5998 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454215 5998 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454218 5998 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454221 5998 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454224 5998 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454227 5998 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454230 5998 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454234 5998 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454237 5998 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454239 5998 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454242 5998 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454245 5998 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454247 5998 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454251 5998 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:29:30.455565 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454254 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454256 5998 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454259 5998 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454262 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454265 5998 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454268 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454271 5998 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454274 5998 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454277 5998 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454279 5998 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454282 5998 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454285 5998 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454287 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454290 5998 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454293 5998 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454296 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454299 5998 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454302 5998 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454304 5998 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454307 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:29:30.456049 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454310 5998 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:29:30.456547 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454314 5998 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:29:30.456547 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454317 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:29:30.456547 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454320 5998 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:29:30.456547 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454323 5998 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:29:30.456547 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454328 5998 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:29:30.456547 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.454332 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:29:30.456547 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455115 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:29:30.456547 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455122 5998 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:29:30.456547 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455126 5998 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:29:30.456547 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455128 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:29:30.456547 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455131 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:29:30.456547 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455134 5998 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:29:30.456547 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455139 5998 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:29:30.456547 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455143 5998 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:29:30.456547 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455147 5998 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:29:30.456547 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455151 5998 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:29:30.456547 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455153 5998 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:29:30.456547 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455156 5998 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455160 5998 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455163 5998 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455166 5998 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455168 5998 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455171 5998 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455174 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455177 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455179 5998 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455182 5998 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455185 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455187 5998 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455190 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455193 5998 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455195 5998 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455198 5998 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455201 5998 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455204 5998 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455207 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455209 5998 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:29:30.457023 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455212 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455218 5998 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455221 5998 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455224 5998 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455226 5998 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455229 5998 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455231 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455234 5998 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455237 5998 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455240 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455243 5998 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455246 5998 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455249 5998 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455252 5998 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455254 5998 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455257 5998 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455260 5998 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455262 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455265 5998 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455268 5998 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:29:30.457627 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455271 5998 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455273 5998 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455276 5998 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455279 5998 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455282 5998 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455284 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455287 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455289 5998 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455292 5998 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455295 5998 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455298 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455301 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455303 5998 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455306 5998 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455310 5998 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455312 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455315 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455318 5998 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455321 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:29:30.458144 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455324 5998 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455327 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455330 5998 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455332 5998 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455335 5998 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455337 5998 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455340 5998 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455343 5998 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455345 5998 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455348 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455350 5998 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455353 5998 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455356 5998 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455358 5998 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455361 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455364 5998 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455366 5998 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455369 5998 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455371 5998 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:29:30.458655 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455374 5998 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455377 5998 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455380 5998 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455382 5998 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455385 5998 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455388 5998 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455391 5998 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455395 5998 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455399 5998 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455404 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455408 5998 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455412 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455416 5998 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455420 5998 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455423 5998 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455428 5998 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455432 5998 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455906 5998 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455911 5998 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455914 5998 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:29:30.459187 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455917 5998 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455920 5998 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455923 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455926 5998 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455928 5998 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455931 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455934 5998 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455937 5998 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455939 5998 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455942 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455945 5998 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455948 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455950 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455953 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455956 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455960 5998 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455962 5998 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455965 5998 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455972 5998 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:29:30.459662 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455975 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455977 5998 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455980 5998 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455983 5998 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455986 5998 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455989 5998 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455991 5998 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455995 5998 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.455998 5998 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456001 5998 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456005 5998 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456007 5998 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456010 5998 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456013 5998 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456016 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456019 5998 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456021 5998 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456024 5998 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456027 5998 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456029 5998 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:29:30.460193 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456033 5998 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456035 5998 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456038 5998 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456041 5998 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456043 5998 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456046 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456049 5998 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456051 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456054 5998 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456057 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456061 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456066 5998 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456070 5998 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456073 5998 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456076 5998 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456079 5998 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456082 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456084 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456101 5998 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:29:30.460696 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456103 5998 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456108 5998 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456112 5998 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456115 5998 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456119 5998 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456122 5998 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456125 5998 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456127 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456130 5998 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456133 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456135 5998 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456138 5998 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456141 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456144 5998 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456146 5998 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456149 5998 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456151 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456154 5998 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456157 5998 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:29:30.461252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456160 5998 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456162 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456165 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456167 5998 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456170 5998 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456175 5998 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456178 5998 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456181 5998 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456183 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456186 5998 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456192 5998 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456196 5998 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456198 5998 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456201 5998 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456204 5998 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456206 5998 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456209 5998 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456212 5998 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456214 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456217 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:29:30.461709 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456220 5998 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456222 5998 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456225 5998 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456227 5998 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456230 5998 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.456232 5998 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456329 5998 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456340 5998 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456349 5998 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456356 5998 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456361 5998 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456364 5998 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456368 5998 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456373 5998 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456376 5998 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456379 5998 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456383 5998 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456390 5998 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456393 5998 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456396 5998 flags.go:64] FLAG: --cgroup-root="" Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456399 5998 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456402 5998 flags.go:64] FLAG: --client-ca-file="" Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456405 5998 flags.go:64] FLAG: --cloud-provider="external" Oct 30 01:29:30.462230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456409 5998 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456414 5998 flags.go:64] FLAG: --cluster-domain="" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456417 5998 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456420 5998 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456423 5998 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456427 5998 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456431 5998 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456434 5998 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456437 5998 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456441 5998 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456444 5998 flags.go:64] FLAG: --contention-profiling="false" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456447 5998 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456450 5998 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456454 5998 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456457 5998 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456461 5998 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456464 5998 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456467 5998 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456470 5998 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456473 5998 flags.go:64] FLAG: --enable-server="true" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456476 5998 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456481 5998 flags.go:64] FLAG: --event-burst="100" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456485 5998 flags.go:64] FLAG: --event-qps="50" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456488 5998 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 01:29:30.462800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456491 5998 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456494 5998 flags.go:64] FLAG: --eviction-hard="" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456504 5998 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456507 5998 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456511 5998 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456514 5998 flags.go:64] FLAG: --eviction-soft="" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456517 5998 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456520 5998 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456523 5998 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456527 5998 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456530 5998 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456543 5998 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456549 5998 flags.go:64] FLAG: --feature-gates="" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456560 5998 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456564 5998 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456569 5998 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456574 5998 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456579 5998 flags.go:64] FLAG: --healthz-port="10248" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456583 5998 flags.go:64] FLAG: --help="false" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456586 5998 flags.go:64] FLAG: --hostname-override="ip-10-0-130-0.ec2.internal" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456589 5998 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456592 5998 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456595 5998 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456598 5998 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 01:29:30.463413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456602 5998 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456605 5998 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456608 5998 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456611 5998 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456614 5998 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456617 5998 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456621 5998 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456624 5998 flags.go:64] FLAG: --kube-reserved="" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456627 5998 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456630 5998 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456636 5998 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456639 5998 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456642 5998 flags.go:64] FLAG: --lock-file="" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456645 5998 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456648 5998 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456651 5998 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456657 5998 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456660 5998 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456664 5998 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456667 5998 flags.go:64] FLAG: --logging-format="text" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456670 5998 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456673 5998 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456676 5998 flags.go:64] FLAG: --manifest-url="" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456679 5998 flags.go:64] FLAG: --manifest-url-header="" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456684 5998 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 01:29:30.464037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456687 5998 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456691 5998 flags.go:64] FLAG: --max-pods="110" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456695 5998 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456698 5998 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456701 5998 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456704 5998 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456707 5998 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456710 5998 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456713 5998 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456720 5998 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456723 5998 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456727 5998 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456730 5998 flags.go:64] FLAG: --pod-cidr="" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456733 5998 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456736 5998 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456739 5998 flags.go:64] FLAG: --pods-per-core="0" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456742 5998 flags.go:64] FLAG: --port="10250" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456747 5998 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456751 5998 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04d13b4367f94ae39" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456754 5998 flags.go:64] FLAG: --qos-reserved="" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456757 5998 flags.go:64] FLAG: --read-only-port="10255" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456760 5998 flags.go:64] FLAG: --register-node="true" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456763 5998 flags.go:64] FLAG: --register-with-taints="" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456767 5998 flags.go:64] FLAG: --registry-burst="10" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456771 5998 flags.go:64] FLAG: --registry-qps="5" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456773 5998 flags.go:64] FLAG: --reserved-cpus="" Oct 30 01:29:30.464659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456777 5998 flags.go:64] FLAG: --reserved-memory="" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456781 5998 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456784 5998 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456788 5998 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456791 5998 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456794 5998 flags.go:64] FLAG: --runonce="false" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456797 5998 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456800 5998 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456803 5998 flags.go:64] FLAG: --seccomp-default="false" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456806 5998 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456810 5998 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456813 5998 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456816 5998 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456819 5998 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456822 5998 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456826 5998 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456829 5998 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456831 5998 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456835 5998 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456838 5998 flags.go:64] FLAG: --system-cgroups="" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456841 5998 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456847 5998 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456850 5998 flags.go:64] FLAG: --tls-cert-file="" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456855 5998 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456859 5998 flags.go:64] FLAG: --tls-min-version="" Oct 30 01:29:30.465309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456862 5998 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456865 5998 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456868 5998 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456871 5998 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456874 5998 flags.go:64] FLAG: --v="2" Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456879 5998 flags.go:64] FLAG: --version="false" Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456883 5998 flags.go:64] FLAG: --vmodule="" Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456887 5998 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.456890 5998 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457022 5998 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457026 5998 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457030 5998 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457034 5998 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457037 5998 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457041 5998 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457044 5998 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457047 5998 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457049 5998 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457052 5998 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457055 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457058 5998 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:29:30.465928 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457060 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457063 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457066 5998 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457068 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457071 5998 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457074 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457076 5998 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457079 5998 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457082 5998 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457099 5998 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457102 5998 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457104 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457107 5998 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457110 5998 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457113 5998 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457115 5998 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457118 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457121 5998 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457123 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:29:30.466799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457127 5998 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457130 5998 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457133 5998 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457135 5998 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457138 5998 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457141 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457143 5998 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457146 5998 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457149 5998 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457151 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457154 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457157 5998 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457159 5998 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457162 5998 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457165 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457167 5998 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457170 5998 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457173 5998 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457175 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457178 5998 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:29:30.467480 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457181 5998 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457183 5998 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457187 5998 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457190 5998 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457193 5998 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457196 5998 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457198 5998 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457202 5998 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457206 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457209 5998 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457213 5998 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457217 5998 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457220 5998 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457223 5998 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457226 5998 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457228 5998 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457232 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457234 5998 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457237 5998 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:29:30.467953 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457239 5998 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457242 5998 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457245 5998 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457247 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457250 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457253 5998 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457255 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457258 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457260 5998 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457263 5998 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457266 5998 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457268 5998 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457271 5998 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457274 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457276 5998 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457281 5998 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457283 5998 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457286 5998 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457289 5998 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457291 5998 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:29:30.468451 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457294 5998 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:29:30.469260 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457297 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:29:30.469260 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457300 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:29:30.469260 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457302 5998 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:29:30.469260 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457305 5998 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:29:30.469260 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457308 5998 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:29:30.469260 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457310 5998 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:29:30.469260 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457313 5998 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:29:30.469260 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457317 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:29:30.469260 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457319 5998 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:29:30.469260 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457322 5998 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:29:30.469260 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457325 5998 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:29:30.469260 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457327 5998 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:29:30.469260 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457330 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:29:30.469260 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457333 5998 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:29:30.469260 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457336 5998 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:29:30.469260 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.457362 5998 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457398 5998 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457403 5998 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457406 5998 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457409 5998 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457412 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457414 5998 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457418 5998 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457420 5998 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457423 5998 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457426 5998 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457430 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457433 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457436 5998 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457438 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457441 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457443 5998 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457446 5998 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457449 5998 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457451 5998 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:29:30.469697 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457454 5998 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457457 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457459 5998 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457462 5998 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457465 5998 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457468 5998 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457473 5998 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457476 5998 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457479 5998 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457482 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457484 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457487 5998 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457490 5998 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457492 5998 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457495 5998 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457498 5998 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457501 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457503 5998 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457506 5998 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457508 5998 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:29:30.470388 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457511 5998 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457514 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457517 5998 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457522 5998 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457525 5998 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457528 5998 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457530 5998 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457533 5998 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457536 5998 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457539 5998 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457541 5998 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457544 5998 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457547 5998 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457549 5998 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457552 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457555 5998 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457557 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457560 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457564 5998 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:29:30.470907 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457567 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457570 5998 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457573 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457575 5998 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457578 5998 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457581 5998 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457583 5998 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457586 5998 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457589 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457592 5998 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457594 5998 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457597 5998 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457600 5998 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457603 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457605 5998 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457608 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457613 5998 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457616 5998 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457619 5998 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457621 5998 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:29:30.471429 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457624 5998 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457626 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457629 5998 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457632 5998 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457635 5998 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457638 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457641 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457644 5998 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457646 5998 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457649 5998 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457652 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457656 5998 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457659 5998 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457662 5998 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457665 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457668 5998 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457671 5998 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457673 5998 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457676 5998 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457679 5998 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:29:30.472021 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457682 5998 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:29:30.472799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457684 5998 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:29:30.472799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457687 5998 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:29:30.472799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457691 5998 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:29:30.472799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457695 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:29:30.472799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457697 5998 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:29:30.472799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457703 5998 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:29:30.472799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.457707 5998 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:29:30.472799 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.462108 5998 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 01:29:30.472799 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.462128 5998 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 01:29:30.472799 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.462149 5998 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 01:29:30.472799 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.462154 5998 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 01:29:30.472799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462243 5998 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:29:30.472799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462249 5998 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:29:30.472799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462253 5998 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:29:30.472799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462257 5998 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:29:30.472799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462260 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:29:30.472799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462263 5998 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:29:30.472799 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462266 5998 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462269 5998 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462272 5998 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462275 5998 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462277 5998 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462280 5998 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462288 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462291 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462294 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462297 5998 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462300 5998 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462302 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462306 5998 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462308 5998 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462311 5998 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462314 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462317 5998 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462319 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462322 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:29:30.473350 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462326 5998 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:29:30.473814 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462329 5998 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:29:30.473814 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462331 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:29:30.473814 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462334 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:29:30.473814 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462337 5998 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:29:30.473814 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462340 5998 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:29:30.473814 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462343 5998 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:29:30.473814 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462346 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:29:30.473814 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462348 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:29:30.473814 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462351 5998 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:29:30.473814 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462354 5998 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:29:30.473814 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462365 5998 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:29:30.473814 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462370 5998 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:29:30.473814 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462373 5998 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:29:30.473814 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462376 5998 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:29:30.473814 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462381 5998 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:29:30.473814 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462384 5998 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:29:30.473814 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462387 5998 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:29:30.473814 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462390 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462393 5998 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462401 5998 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462404 5998 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462407 5998 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462410 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462412 5998 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462415 5998 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462418 5998 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462421 5998 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462424 5998 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462426 5998 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462429 5998 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462432 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462435 5998 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462443 5998 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462446 5998 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462449 5998 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462453 5998 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462455 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462458 5998 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:29:30.474269 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462461 5998 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462464 5998 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462467 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462469 5998 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462472 5998 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462475 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462478 5998 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462481 5998 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462483 5998 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462486 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462489 5998 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462491 5998 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462494 5998 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462497 5998 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462505 5998 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462508 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462510 5998 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462513 5998 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462516 5998 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462519 5998 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:29:30.474805 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462522 5998 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462524 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462527 5998 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462530 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462532 5998 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462535 5998 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462538 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462541 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462544 5998 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462546 5998 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462549 5998 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462552 5998 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462554 5998 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462557 5998 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462559 5998 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462562 5998 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462565 5998 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462568 5998 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462570 5998 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:29:30.475308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462573 5998 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:29:30.475809 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462575 5998 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:29:30.475809 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462578 5998 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:29:30.475809 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.462606 5998 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:29:30.475809 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462795 5998 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:29:30.475809 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462799 5998 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:29:30.475809 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462802 5998 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:29:30.475809 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462805 5998 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:29:30.475809 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462813 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:29:30.475809 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462816 5998 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:29:30.475809 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462819 5998 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:29:30.475809 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462822 5998 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:29:30.475809 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462825 5998 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:29:30.475809 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462828 5998 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:29:30.475809 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462831 5998 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:29:30.475809 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462833 5998 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:29:30.475809 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462836 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462839 5998 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462841 5998 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462844 5998 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462847 5998 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462849 5998 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462852 5998 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462855 5998 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462858 5998 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462861 5998 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462863 5998 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462866 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462869 5998 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462872 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462874 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462877 5998 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462880 5998 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462882 5998 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462885 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462887 5998 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:29:30.476356 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462890 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462892 5998 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462895 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462898 5998 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462901 5998 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462909 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462911 5998 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462914 5998 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462917 5998 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462919 5998 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462922 5998 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462925 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462928 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462931 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462933 5998 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462936 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462939 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462941 5998 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462944 5998 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:29:30.476873 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462948 5998 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462951 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462954 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462957 5998 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462960 5998 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462963 5998 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462965 5998 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462968 5998 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462971 5998 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462974 5998 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462976 5998 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462979 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462981 5998 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462984 5998 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462987 5998 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462989 5998 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462992 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462995 5998 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.462997 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463005 5998 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:29:30.477340 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463008 5998 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463011 5998 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463013 5998 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463016 5998 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463020 5998 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463024 5998 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463027 5998 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463035 5998 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463038 5998 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463041 5998 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463044 5998 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463047 5998 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463050 5998 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463053 5998 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463055 5998 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463058 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463061 5998 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463064 5998 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463067 5998 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:29:30.477863 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463069 5998 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:29:30.478349 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463072 5998 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:29:30.478349 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463075 5998 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:29:30.478349 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463077 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:29:30.478349 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463080 5998 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:29:30.478349 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463083 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:29:30.478349 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463085 5998 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:29:30.478349 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463102 5998 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:29:30.478349 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463107 5998 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:29:30.478349 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463111 5998 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:29:30.478349 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463114 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:29:30.478349 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463117 5998 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:29:30.478349 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463120 5998 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:29:30.478349 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463130 5998 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:29:30.478349 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463135 5998 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:29:30.478349 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463139 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:29:30.478349 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.463168 5998 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463242 5998 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463247 5998 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463250 5998 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463253 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463257 5998 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463260 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463263 5998 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463266 5998 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463269 5998 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463271 5998 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463274 5998 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463278 5998 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463284 5998 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463290 5998 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463295 5998 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463299 5998 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463301 5998 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463304 5998 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463307 5998 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:29:30.478744 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463310 5998 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463313 5998 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463315 5998 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463318 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463321 5998 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463324 5998 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463326 5998 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463330 5998 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463335 5998 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463339 5998 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463347 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463350 5998 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463353 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463358 5998 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463362 5998 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463366 5998 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463370 5998 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463376 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463380 5998 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463382 5998 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:29:30.479258 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463385 5998 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463388 5998 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463391 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463394 5998 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463396 5998 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463399 5998 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463402 5998 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463404 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463407 5998 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463410 5998 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463413 5998 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463415 5998 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463418 5998 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463421 5998 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463425 5998 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463428 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463430 5998 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463433 5998 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463436 5998 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463438 5998 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:29:30.479790 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463441 5998 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463444 5998 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463446 5998 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463455 5998 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463459 5998 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463464 5998 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463467 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463469 5998 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463472 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463484 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463487 5998 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463490 5998 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463493 5998 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463496 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463498 5998 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463501 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463504 5998 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463506 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463509 5998 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463512 5998 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:29:30.480301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463514 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:29:30.480793 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463517 5998 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:29:30.480793 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463520 5998 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:29:30.480793 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463630 5998 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:29:30.480793 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463636 5998 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:29:30.480793 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463639 5998 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:29:30.480793 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463643 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:29:30.480793 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463646 5998 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:29:30.480793 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463649 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:29:30.480793 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463651 5998 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:29:30.480793 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463654 5998 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:29:30.480793 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463657 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:29:30.480793 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463659 5998 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:29:30.480793 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463662 5998 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:29:30.480793 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463667 5998 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:29:30.480793 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463673 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:29:30.480793 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463687 5998 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:29:30.480793 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463692 5998 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:29:30.480793 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463696 5998 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:29:30.481308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463699 5998 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:29:30.481308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463702 5998 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:29:30.481308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463705 5998 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:29:30.481308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463709 5998 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:29:30.481308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463712 5998 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:29:30.481308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463715 5998 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:29:30.481308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463718 5998 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:29:30.481308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:30.463720 5998 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:29:30.481308 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.463786 5998 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 01:29:30.481308 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.464778 5998 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 01:29:30.481308 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.464833 5998 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 01:29:30.481308 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.465038 5998 server.go:1014] "Starting client certificate rotation" Oct 30 01:29:30.481308 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.465183 5998 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 01:29:30.481308 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.465231 5998 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 01:23:19 +0000 UTC" deadline="2028-04-20 06:03:50.143566007 +0000 UTC" Oct 30 01:29:30.481308 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.465262 5998 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12916h34m19.678305843s" Oct 30 01:29:30.481308 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.465556 5998 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:29:30.481693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.468118 5998 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:29:30.481693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.469554 5998 log.go:25] "Validated CRI v1 runtime API" Oct 30 01:29:30.481693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.471386 5998 log.go:25] "Validated CRI v1 image API" Oct 30 01:29:30.481693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.472206 5998 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 01:29:30.481693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.474251 5998 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 95bd4926-4597-4527-8411-f8edc5a3c90e:/dev/nvme0n1p3 ee1a38bd-3918-482f-acad-028e5cfeebb0:/dev/nvme0n1p4] Oct 30 01:29:30.481857 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.474269 5998 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15a65d279b86c4bb5028f6df9dd3f842f2fe926c1454b8656321728f4455f3b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15a65d279b86c4bb5028f6df9dd3f842f2fe926c1454b8656321728f4455f3b4/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b9a49f978b8703acca46ee6d0b3da0bcffee1ff4f686b59538aeac1d97ab664/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b9a49f978b8703acca46ee6d0b3da0bcffee1ff4f686b59538aeac1d97ab664/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d7fdd368334d11a5e39d01f71bdac4b4c5de1013b365826129fe7082e083759/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d7fdd368334d11a5e39d01f71bdac4b4c5de1013b365826129fe7082e083759/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27371582628e07d070fe27c81ce4db4d80439cbe02c18b552ebb7dffab3d915b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27371582628e07d070fe27c81ce4db4d80439cbe02c18b552ebb7dffab3d915b/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d9c51ec0745adf7780cd055fbbbcf067c6bc5a9a4f987ec6e3ead8258ce72ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d9c51ec0745adf7780cd055fbbbcf067c6bc5a9a4f987ec6e3ead8258ce72ca/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30185886dd6355130b98e7f2717a94945ff17d24f2d204ed8bf3d26c2fd6cfcf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30185886dd6355130b98e7f2717a94945ff17d24f2d204ed8bf3d26c2fd6cfcf/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3daab102c6828b9bfd879686ef3bc503036ede472096f62ebe89b7894bf61417/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3daab102c6828b9bfd879686ef3bc503036ede472096f62ebe89b7894bf61417/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f185188f4a22d5bd7e0b9164ff1701202c2eef5e35dfbf8fd8d004879e97812/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f185188f4a22d5bd7e0b9164ff1701202c2eef5e35dfbf8fd8d004879e97812/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/637002c490455d177e3c95d7933023979dc9946e4ffe05bb08f8e03c5d0b92d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/637002c490455d177e3c95d7933023979dc9946e4ffe05bb08f8e03c5d0b92d7/userdata/shm major:0 minor:398 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73d90759d94928512bd548cf432ea46ffb8eab3daea096ef0d84adb8764c6ac8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73d90759d94928512bd548cf432ea46ffb8eab3daea096ef0d84adb8764c6ac8/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f1b5cde92f5e98099f7b8b4e3ce0aa5a8f19a3bdacfa18e601adbc56d9ca63b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f1b5cde92f5e98099f7b8b4e3ce0aa5a8f19a3bdacfa18e601adbc56d9ca63b/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b47ad09bf1c892f8f126aba288c1e4c57b17c01d77a7fff7850ee48b39bf2ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b47ad09bf1c892f8f126aba288c1e4c57b17c01d77a7fff7850ee48b39bf2ce/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9edf2b9801239a3c71c8e460fc7a1de9fc4ceefb1af1d5affc9bcd50e7423876/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9edf2b9801239a3c71c8e460fc7a1de9fc4ceefb1af1d5affc9bcd50e7423876/userdata/shm major:0 minor:305 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be9d7b5fa48523697c0914953d02bd598c0b0a4d40650a9a1edc5964b8372ad7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be9d7b5fa48523697c0914953d02bd598c0b0a4d40650a9a1edc5964b8372ad7/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4320bb0b198f2a9176fd4a26532dc30e402064783fc6444fd2f4efc57ab46bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4320bb0b198f2a9176fd4a26532dc30e402064783fc6444fd2f4efc57ab46bc/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfa1331afcebcca333f0e9ed66f18f2c3d5f10166676c32c7f6642aa5949d5c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfa1331afcebcca333f0e9ed66f18f2c3d5f10166676c32c7f6642aa5949d5c5/userdata/shm major:0 minor:395 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de46b4224869ff221f36081f4ff9119a601df3ef83012041978fcab90cafee19/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de46b4224869ff221f36081f4ff9119a601df3ef83012041978fcab90cafee19/userdata/shm major:0 minor:358 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec7aa30a140f804d41a551deb79a6db69982d52daedfe91b2a33a41c0470c447/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec7aa30a140f804d41a551deb79a6db69982d52daedfe91b2a33a41c0470c447/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec810a6918efaf59b46323d1b76630d9d0495662bed0d79cd8f83a6a10330e34/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec810a6918efaf59b46323d1b76630d9d0495662bed0d79cd8f83a6a10330e34/userdata/shm major:0 minor:297 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/edbaaa39931f564c22987e5ef6df31a82673dffe17e97708e4661be6d19b523b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/edbaaa39931f564c22987e5ef6df31a82673dffe17e97708e4661be6d19b523b/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe7a5c1e60ceb682069567715a19487b2a4e4e34db069de628afd3114e733839/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe7a5c1e60ceb682069567715a19487b2a4e4e34db069de628afd3114e733839/userdata/shm major:0 minor:400 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4/volumes/kubernetes.io~projected/kube-api-access-qgnfn:{mountpoint:/var/lib/kubelet/pods/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4/volumes/kubernetes.io~projected/kube-api-access-qgnfn major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/174be2db-9c13-45fe-b91f-c3302ecb57e2/volumes/kubernetes.io~projected/kube-api-access-jjjjr:{mountpoint:/var/lib/kubelet/pods/174be2db-9c13-45fe-b91f-c3302ecb57e2/volumes/kubernetes.io~projected/kube-api-access-jjjjr major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18cee84e-69dd-4296-98d7-de2e6e0c6fe0/volumes/kubernetes.io~projected/kube-api-access-dbxj6:{mountpoint:/var/lib/kubelet/pods/18cee84e-69dd-4296-98d7-de2e6e0c6fe0/volumes/kubernetes.io~projected/kube-api-access-dbxj6 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3657b83a-1cdf-44de-9a9d-2800cfae5524/volumes/kubernetes.io~projected/kube-api-access-xtbks:{mountpoint:/var/lib/kubelet/pods/3657b83a-1cdf-44de-9a9d-2800cfae5524/volumes/kubernetes.io~projected/kube-api-access-xtbks major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3657b83a-1cdf-44de-9a9d-2800cfae5524/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/3657b83a-1cdf-44de-9a9d-2800cfae5524/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3657b83a-1cdf-44de-9a9d-2800cfae5524/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/3657b83a-1cdf-44de-9a9d-2800cfae5524/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3657b83a-1cdf-44de-9a9d-2800cfae5524/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/3657b83a-1cdf-44de-9a9d-2800cfae5524/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40d2d25c-e7a0-4ded-9674-19c6d371522e/volumes/kubernetes.io~projected/kube-api-access-qwzmn:{mountpoint:/var/lib/kubelet/pods/40d2d25c-e7a0-4ded-9674-19c6d371522e/volumes/kubernetes.io~projected/kube-api-access-qwzmn major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43e8ec8c-149e-4257-a891-7b117a79c319/volumes/kubernetes.io~projected/kube-api-access-ssfgz:{mountpoint:/var/lib/kubelet/pods/43e8ec8c-149e-4257-a891-7b117a79c319/volumes/kubernetes.io~projected/kube-api-access-ssfgz major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43e8ec8c-149e-4257-a891-7b117a79c319/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/43e8ec8c-149e-4257-a891-7b117a79c319/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43e8ec8c-149e-4257-a891-7b117a79c319/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/43e8ec8c-149e-4257-a891-7b117a79c319/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f486e9e-aa32-4a9a-877b-eca8dbba7091/volumes/kubernetes.io~projected/kube-api-access-wvtjv:{mountpoint:/var/lib/kubelet/pods/4f486e9e-aa32-4a9a-877b-eca8dbba7091/volumes/kubernetes.io~projected/kube-api-access-wvtjv major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~projected/kube-api-access-n55vv:{mountpoint:/var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~projected/kube-api-access-n55vv major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ff44b73-9f64-4fca-95f8-6a5a24540843/volumes/kubernetes.io~projected/kube-api-access-kwgtk:{mountpoint:/var/lib/kubelet/pods/6ff44b73-9f64-4fca-95f8-6a5a24540843/volumes/kubernetes.io~projected/kube-api-access-kwgtk major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ff44b73-9f64-4fca-95f8-6a5a24540843/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/6ff44b73-9f64-4fca-95f8-6a5a24540843/volumes/kubernetes.io~secret/metrics-certs major:0 minor:390 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8af570c7-62ff-4f33-93df-1714b5ffc20c/volumes/kubernetes.io~projected/kube-api-access-xqrp9:{mountpoint:/var/lib/kubelet/pods/8af570c7-62ff-4f33-93df-1714b5ffc20c/volumes/kubernetes.io~projected/kube-api-access-xqrp9 major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volume-subpaths/web-config/config-reloader/2 major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volume-subpaths/web-config/prometheus/15 major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~empty-dir/config-out major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~projected/kube-api-access-wspxg:{mountpoint:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~projected/kube-api-access-wspxg major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~projected/tls-assets major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/config major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/web-config major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91d108c9-4e8c-4d00-8e42-1eecd52741a2/volumes/kubernetes.io~projected/kube-api-access-2hwlz:{mountpoint:/var/lib/kubelet/pods/91d108c9-4e8c-4d00-8e42-1eecd52741a2/volumes/kubernetes.io~projected/kube-api-access-2hwlz major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91d108c9-4e8c-4d00-8e42-1eecd52741a2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/91d108c9-4e8c-4d00-8e42-1eecd52741a2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0d9a754-4e82-4dd5-b423-50624589bdda/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a0d9a754-4e82-4dd5-b423-50624589bdda/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4c33058-34c5-49f3-b803-984c54ff5dad/volumes/kubernetes.io~projected/kube-api-access-zqpgm:{mountpoint:/var/lib/kubelet/pods/b4c33058-34c5-49f3-b803-984c54ff5dad/volumes/kubernetes.io~projected/kube-api-access-zqpgm major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4c33058-34c5-49f3-b803-984c54ff5dad/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b4c33058-34c5-49f3-b803-984c54ff5dad/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4c33058-34c5-49f3-b803-984c54ff5dad/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/b4c33058-34c5-49f3-b803-984c54ff5dad/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bde30efc-28f6-4fb1-83b7-2b59d6d34359/volumes/kubernetes.io~projected/kube-api-access-wd6sl:{mountpoint:/var/lib/kubelet/pods/bde30efc-28f6-4fb1-83b7-2b59d6d34359/volumes/kubernetes.io~projected/kube-api-access-wd6sl major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bde30efc-28f6-4fb1-83b7-2b59d6d34359/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/bde30efc-28f6-4fb1-83b7-2b59d6d34359/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c01f6b4a-5666-4734-b403-3818692c3a74/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c01f6b4a-5666-4734-b403-3818692c3a74/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c01f6b4a-5666-4734-b403-3818692c3a74/volumes/kubernetes.io~projected/kube-api-access-kkjlt:{mountpoint:/var/lib/kubelet/pods/c01f6b4a-5666-4734-b403-3818692c3a74/volumes/kubernetes.io~projected/kube-api-access-kkjlt major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c01f6b4a-5666-4734-b403-3818692c3a74/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c01f6b4a-5666-4734-b403-3818692c3a74/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1facbe7-deb0-4fec-94de-42a0dd91ff4f/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d1facbe7-deb0-4fec-94de-42a0dd91ff4f/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1facbe7-deb0-4fec-94de-42a0dd91ff4f/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d1facbe7-deb0-4fec-94de-42a0dd91ff4f/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1facbe7-deb0-4fec-94de-42a0dd91ff4f/volumes/kubernetes.io~projected/kube-api-access-vtwhw:{mountpoint:/var/lib/kubelet/pods/d1facbe7-deb0-4fec-94de-42a0dd91ff4f/volumes/kubernetes.io~projected/kube-api-access-vtwhw major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d50883a3-5056-4826-884e-90fe75441e5d/volumes/kubernetes.io~projected/kube-api-access-gwl72:{mountpoint:/var/lib/kubelet/pods/d50883a3-5056-4826-884e-90fe75441e5d/volumes/kubernetes.io~projected/kube-api-access-gwl72 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da638295-de7f-462a-9a14-2b8105e636b6/volumes/kubernetes.io~projected/kube-api-access-scl9k:{mountpoint:/var/lib/kubelet/pods/da638295-de7f-462a-9a14-2b8105e636b6/volumes/kubernetes.io~projected/kube-api-access-scl9k major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da638295-de7f-462a-9a14-2b8105e636b6/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/da638295-de7f-462a-9a14-2b8105e636b6/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:394 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/89fa4d0a02fef0f00f1410e35bb7731fb1b36ea7b46f5c0dba55323015fbcb37/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6567a51f19d9ca296e7d0f2d6b5219edb9b6729c9c3b4feb28a06d2ecc6a0d0e/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/070cde865a6ddf034558a0ff500fd7f502d35aee5876d47b45662e2920378e6a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/084a27abee1d4ced7cc19bc6bbd250f81eeb4e625bc538fd9332a84ded395c65/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/4042c9f3fadc0e4fe33a61e3dad0400017595a7101f15f9d6b5511817f8cdfdb/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/7e2e6b96480beff21340f42a711871ec318f09ebe17f1738c3dcb2c7af3c6d77/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/10bbf4e60549d19b27c52c6aa7aefc2645dabcdd2f911c408703bbbfbd72dd33/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/cb21cdd567a680e87b2de44b28b620fd81b5aea18097a602510d89d1eedbfdd9/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/2c0dde6a37b9e5abb8c6505d78bd480eb479e5b3b7e7fa8a346ef657c4f8c38c/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/f9d7e917a518a7a2ef75a1f10ff0ae3f28f84b190e877eceab51539b03311841/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/4215f964c6e1a851e83eed0a794f4ab8a6b65889af5ff1255f7c409658f34344/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/882c8d464f5e04f42b53e73a948c962f18ce9b4a8b67849e00d39f5a66c39c4b/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6f6eb16ae3273932604ff9bffd982c0233f2830ef5263acd66763746c5b97e4b/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/fd37d1e2c48b667353c4d766f73fd1b6272dc49d06d1b5ddcc331d11bac7f674/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/026ce986b903879b07f96cb65d91abd14cfb1cda788dd05fec960a8c5d3508e2/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/882bedf22b4a5c0634e84ddee36d291fc5801c22b4d9a545edbac55efa9229f9/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ab9f372689ca5be44af78d22ed7f4cf4586d0c2fcb7ee3cffda3a280ec05ca96/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/69656294d0eb8a5aee06f7db9767e36272a1c0fffb1902908c35638ec5d4fac2/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/fc1c110898ccf3a980179e1f17a5a365949b033483d28b578122687ed50e1878/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/3b86b1627a493f2d322609f90cf38d1e52c986067fbb2af5ee8d8e2295bb5af3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7b58ddf3869bfaa084faa29c44f13ac1b25673bb885969fe18eadfa55e3631b6/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/d678ce47126b29450dbb6141a59b3db1011dd18731c73600c9ab368a1637b4e3/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/5dd7cbcba8b9a9915f6cdd5911ae8eec6d197dec4a557aab634a194d913a24a2/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/1ecec8cf8da2da6393e35aee697773ef7fd18971c2ba695f68e61fc8ed627bc7/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/031cbeeda244c730c117a714dc57a6eb3d8da5961cf08fed7fba6abfd1736be2/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/89c32f8a3bd5eaa6bf1e83ff6fdf4591cb6dd15ebf0d68e1292f1a20d0b2bb09/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/421fcec0c69f912f5aa4d6476992d84c80831a33810c3ee95ad5a8f3c51b9d55/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/d168718e5c62e82445145dc25f5147334334ae70cd23de32c8290b61ad3c60c1/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/776e87e817ed8a3fe4c56d722a85d934903c1fdcb9428bdf534cfbc60d5e1203/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/6403642f662323230a65a63e69c64cab404b304ddfc26645f34a7b4b29304feb/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/cbc2d6ab8186df7489492fa93f452f35ed44e6431991818cb41cdab841eacdc2/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/ffb86b9a17b94fac8cdb438157537f47d7d4e8ac866363424841848a31a50155/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/b17e10105b689fb1fc3ec93eac051f7e86a850d14eb43c82ad23bfcb36acf5e5/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/ac72ce7a75d46ce3850c21382b92af2296e309707062f595e81229dcf58d6db5/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/ab4ac5315e2169570e39704ab2e578e22b7ddca5fb967de324365fa1747db73f/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/dc4483fb596f3999fc5f89b7b1b50f2ad3adba4be157b5f8cbce8a37ee3e088d/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/264aea0dc21a2dfacd7faef498c5f8418d8d5b623f80f90083f630ce43eb41c1/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/29366a3ec728de5a437b7e9a4ac79bf03d4a9d3d72bb6d43611034fffb57450d/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/474bfbb26394973b7f98f55890f357d77031227cd9d2a28b742fa8340090c42c/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/7d0f432bb0713a46a349ad20cd68f588961746fd8662cbf89dbae90d0f48c586/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/d61eea8fe305922c3675332a44da41ca354627e597c318d9781601701c093c2d/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/3602937ffc75ea1eb632a3de87754de3fd3bbcfa7767d276e7ed9743f40d14f9/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/4be36eeaa7f91852f0d1697f8e0ac67d8c9b758d330952d8adfe54fc8a321ffa/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/bfb2184488fcdf4bd7d45b15c5cee8f8f87053b3b3f6850e9f063806004215b1/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/f78ebaf3e5efe856c65db49391267fcfbdbd3e3621b893a9646711e274e2e5b0/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/2e6dc7c480dcb5b01cd5ad80ed38eebbac655c1d194bb3bbf8537354601ed320/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/2fdf42884540b42a9c650a5e334e22ad843a56b20f74698c8923c9372e6bc5c1/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/49714eb01d162f630c50531fd76209f6bb67fcd5811098d3b47b53e90ae5c3c3/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/d119f42c408336350544a4c9041511ceef90747adf79099628b86d3a9eecae34/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/5fb27f1ecb0397241c8320e3bc0c92c5b3e31a63bd7c14ced68ba4698be05274/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/67c0a9f1b9f761139e5806706533d7903f084e55ec88890439cfe01824d453fe/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/a83c6e61d6a994dae2031392d49e550f1d15b1bb0ffcc6d1853da689c9b36b44/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/9a8c98beaf4c7fc24a2ddccba841450078f2435fabe15289447436965912c514/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/fee8a3423863ddb4a1e9ae36ae07bae037218a3af5a34e70575358361b190e60/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/96672225621a68435d536a696509446732a0f1488b757c9a2ee97e688ec05f96/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/027f38d7aebdb6b86b89c9012cb927afb3524d2c486e7e78e662a828a553f10a/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/97aef8e050b196bd904909ed9ffbf9e7fa74fdb4041a850c1ea84baf47e1f7e1/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/7a7f96398c40f818ad8345fb9c29f3ba623df172ca685aa45d642b4b672a3923/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/f09c41ded4928b07b55314dd04f2e9c1a428308cbe70426bef6e76f7d0c91d58/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/0a623b0b488498161afb67872e31a0ac6074bb411ca1def3c7aaf25e454b6c98/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/0bf1d0262ed169e69b24ef622c2813d249a53e2b4b34e593a44cf00bc6110995/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/0ee291892b2ec883c307489546c71570a1cb057f1d9f180626f74f140a044afe/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/af02ff6d5c762a67f543c68f6d6775520cdadb4680b814b430b3415b18b64501/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/656e9b09f0b4d40cdf641c2f03b8751b29604691de0408bfd9e36c2dc16881d5/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/863a2f34de671aacc51ed448b76d6162d61c3a875d033b872e468e53d6a0c1f6/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/b97c7bf095052f34e98cb31fa54af18c16a0d3ac4b9d72e2ec2ebc810ba2910f/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/8164f8553ed7b0a5e0c51b4390422fde6ac605206924b8ec4e16baf909e34a8f/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/550c9ae1669f7a7dab2f15d453439fb2dd50e253f72226d57fa59f4aef24f48a/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a606710eaede77baedd01dd02271766352538657d9a41a8de4c2a7f49ca63d41/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f431aac1c810c5614f2906f3fec07c3b418cc2e9a4e7337ac2aa04607648fe6b/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 01:29:30.482285 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.481898 5998 manager.go:217] Machine: {Timestamp:2026-10-30 01:29:30.481423404 +0000 UTC m=+0.060214476 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097852 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20bd49eca7107ded8fd169f2678842 SystemUUID:ec20bd49-eca7-107d-ed8f-d169f2678842 BootID:c09f8dcd-a2a9-416b-98f0-f03c48b6945f Filesystems:[{Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15a65d279b86c4bb5028f6df9dd3f842f2fe926c1454b8656321728f4455f3b4/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30185886dd6355130b98e7f2717a94945ff17d24f2d204ed8bf3d26c2fd6cfcf/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:291 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:355 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:353 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d7fdd368334d11a5e39d01f71bdac4b4c5de1013b365826129fe7082e083759/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0d9a754-4e82-4dd5-b423-50624589bdda/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:294 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f1b5cde92f5e98099f7b8b4e3ce0aa5a8f19a3bdacfa18e601adbc56d9ca63b/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:352 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c01f6b4a-5666-4734-b403-3818692c3a74/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4/volumes/kubernetes.io~projected/kube-api-access-qgnfn DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d9c51ec0745adf7780cd055fbbbcf067c6bc5a9a4f987ec6e3ead8258ce72ca/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/43e8ec8c-149e-4257-a891-7b117a79c319/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:230 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~projected/kube-api-access-wspxg DeviceMajor:0 DeviceMinor:357 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b4c33058-34c5-49f3-b803-984c54ff5dad/volumes/kubernetes.io~projected/kube-api-access-zqpgm DeviceMajor:0 DeviceMinor:233 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:349 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ff44b73-9f64-4fca-95f8-6a5a24540843/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:390 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:296 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91d108c9-4e8c-4d00-8e42-1eecd52741a2/volumes/kubernetes.io~projected/kube-api-access-2hwlz DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/43e8ec8c-149e-4257-a891-7b117a79c319/volumes/kubernetes.io~projected/kube-api-access-ssfgz DeviceMajor:0 DeviceMinor:232 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:347 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:354 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3657b83a-1cdf-44de-9a9d-2800cfae5524/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:302 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d1facbe7-deb0-4fec-94de-42a0dd91ff4f/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b4c33058-34c5-49f3-b803-984c54ff5dad/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:356 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9edf2b9801239a3c71c8e460fc7a1de9fc4ceefb1af1d5affc9bcd50e7423876/userdata/shm DeviceMajor:0 DeviceMinor:305 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:292 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/18cee84e-69dd-4296-98d7-de2e6e0c6fe0/volumes/kubernetes.io~projected/kube-api-access-dbxj6 DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d50883a3-5056-4826-884e-90fe75441e5d/volumes/kubernetes.io~projected/kube-api-access-gwl72 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d1facbe7-deb0-4fec-94de-42a0dd91ff4f/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ff44b73-9f64-4fca-95f8-6a5a24540843/volumes/kubernetes.io~projected/kube-api-access-kwgtk DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~projected/kube-api-access-n55vv DeviceMajor:0 DeviceMinor:286 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3657b83a-1cdf-44de-9a9d-2800cfae5524/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:303 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:343 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be9d7b5fa48523697c0914953d02bd598c0b0a4d40650a9a1edc5964b8372ad7/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3657b83a-1cdf-44de-9a9d-2800cfae5524/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:301 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8af570c7-62ff-4f33-93df-1714b5ffc20c/volumes/kubernetes.io~projected/kube-api-access-xqrp9 DeviceMajor:0 DeviceMinor:397 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/40d2d25c-e7a0-4ded-9674-19c6d371522e/volumes/kubernetes.io~projected/kube-api-access-qwzmn DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/edbaaa39931f564c22987e5ef6df31a82673dffe17e97708e4661be6d19b523b/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91d108c9-4e8c-4d00-8e42-1eecd52741a2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f486e9e-aa32-4a9a-877b-eca8dbba7091/volumes/kubernetes.io~projected/kube-api-access-wvtjv DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b9a49f978b8703acca46ee6d0b3da0bcffee1ff4f686b59538aeac1d97ab664/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:348 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:394 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c01f6b4a-5666-4734-b403-3818692c3a74/volumes/kubernetes.io~projected/kube-api-access-kkjlt DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b47ad09bf1c892f8f126aba288c1e4c57b17c01d77a7fff7850ee48b39bf2ce/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da638295-de7f-462a-9a14-2b8105e636b6/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bde30efc-28f6-4fb1-83b7-2b59d6d34359/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:295 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27371582628e07d070fe27c81ce4db4d80439cbe02c18b552ebb7dffab3d915b/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:351 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe7a5c1e60ceb682069567715a19487b2a4e4e34db069de628afd3114e733839/userdata/shm DeviceMajor:0 DeviceMinor:400 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec7aa30a140f804d41a551deb79a6db69982d52daedfe91b2a33a41c0470c447/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4fa1c18e-5933-4e09-85f8-28033e6cc4df/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:293 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:351 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:351 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3daab102c6828b9bfd879686ef3bc503036ede472096f62ebe89b7894bf61417/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73d90759d94928512bd548cf432ea46ffb8eab3daea096ef0d84adb8764c6ac8/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da638295-de7f-462a-9a14-2b8105e636b6/volumes/kubernetes.io~projected/kube-api-access-scl9k DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/43e8ec8c-149e-4257-a891-7b117a79c319/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:231 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4320bb0b198f2a9176fd4a26532dc30e402064783fc6444fd2f4efc57ab46bc/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3657b83a-1cdf-44de-9a9d-2800cfae5524/volumes/kubernetes.io~projected/kube-api-access-xtbks DeviceMajor:0 DeviceMinor:304 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de46b4224869ff221f36081f4ff9119a601df3ef83012041978fcab90cafee19/userdata/shm DeviceMajor:0 DeviceMinor:358 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4c33058-34c5-49f3-b803-984c54ff5dad/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec810a6918efaf59b46323d1b76630d9d0495662bed0d79cd8f83a6a10330e34/userdata/shm DeviceMajor:0 DeviceMinor:297 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cfa1331afcebcca333f0e9ed66f18f2c3d5f10166676c32c7f6642aa5949d5c5/userdata/shm DeviceMajor:0 DeviceMinor:395 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c01f6b4a-5666-4734-b403-3818692c3a74/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/bde30efc-28f6-4fb1-83b7-2b59d6d34359/volumes/kubernetes.io~projected/kube-api-access-wd6sl DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1facbe7-deb0-4fec-94de-42a0dd91ff4f/volumes/kubernetes.io~projected/kube-api-access-vtwhw DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f185188f4a22d5bd7e0b9164ff1701202c2eef5e35dfbf8fd8d004879e97812/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:351 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/174be2db-9c13-45fe-b91f-c3302ecb57e2/volumes/kubernetes.io~projected/kube-api-access-jjjjr DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:350 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/637002c490455d177e3c95d7933023979dc9946e4ffe05bb08f8e03c5d0b92d7/userdata/shm DeviceMajor:0 DeviceMinor:398 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:27371582628e07d MacAddress:66:91:2a:3a:3c:20 Speed:10000 Mtu:8901} {Name:637002c490455d1 MacAddress:7a:60:f1:1b:8c:1d Speed:10000 Mtu:8901} {Name:9edf2b9801239a3 MacAddress:66:96:6e:a8:50:5a Speed:10000 Mtu:8901} {Name:be9d7b5fa485236 MacAddress:6e:e8:b7:12:3a:ad Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f6:42:39:55 Speed:0 Mtu:9001} {Name:br-int MacAddress:e6:cd:9e:a9:74:eb Speed:0 Mtu:8901} {Name:cfa1331afcebcca MacAddress:36:49:46:07:84:be Speed:10000 Mtu:8901} {Name:de46b4224869ff2 MacAddress:8e:63:52:f7:07:5e Speed:10000 Mtu:8901} {Name:ec7aa30a140f804 MacAddress:c2:3b:a4:e6:9f:69 Speed:10000 Mtu:8901} {Name:ec810a6918efaf5 MacAddress:0e:9e:65:8f:e9:06 Speed:10000 Mtu:8901} {Name:edbaaa39931f564 MacAddress:6e:82:bb:1a:21:2a Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f6:42:39:55 Speed:0 Mtu:9001} {Name:fe7a5c1e60ceb68 MacAddress:d6:a0:f7:47:a0:c0 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:96:14:11:db:50:26 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1e:65:8b:a0:0e:0b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 01:29:30.482492 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.482289 5998 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 01:29:30.482492 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.482333 5998 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 01:29:30.482718 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.482697 5998 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 01:29:30.482853 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.482718 5998 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-0.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 01:29:30.482895 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.482855 5998 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 01:29:30.482895 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.482864 5998 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 01:29:30.482895 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.482878 5998 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:29:30.482978 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.482904 5998 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:29:30.483010 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.482982 5998 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 01:29:30.483171 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.483159 5998 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 01:29:30.483308 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.483298 5998 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 01:29:30.483354 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.483333 5998 kubelet.go:493] "Attempting to sync node with API server" Oct 30 01:29:30.483354 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.483343 5998 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 01:29:30.483354 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.483353 5998 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 01:29:30.483428 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.483362 5998 kubelet.go:405] "Adding apiserver pod source" Oct 30 01:29:30.483428 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.483372 5998 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 01:29:30.484073 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.484052 5998 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 01:29:30.484073 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.484067 5998 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:29:30.485695 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.484716 5998 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 01:29:30.488084 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488062 5998 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 01:29:30.488084 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488078 5998 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:29:30.488205 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488185 5998 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 01:29:30.488382 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488370 5998 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 01:29:30.488414 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488400 5998 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 01:29:30.488501 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488491 5998 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 01:29:30.488530 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488506 5998 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 01:29:30.488530 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488512 5998 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 01:29:30.488530 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488518 5998 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 01:29:30.488530 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488524 5998 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 01:29:30.488530 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488529 5998 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 01:29:30.488659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488536 5998 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 01:29:30.488659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488542 5998 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 01:29:30.488659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488548 5998 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 01:29:30.488659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488554 5998 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 01:29:30.488659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488563 5998 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 01:29:30.488659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488573 5998 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 01:29:30.488659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488600 5998 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 01:29:30.488659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.488606 5998 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 01:29:30.492070 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.492058 5998 server.go:1263] "Started kubelet" Oct 30 01:29:30.492181 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.492133 5998 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 01:29:30.497698 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.492220 5998 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 01:29:30.497698 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.492345 5998 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 01:29:30.497698 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.492654 5998 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 01:29:30.497698 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.493510 5998 server.go:317] "Adding debug handlers to kubelet server" Oct 30 01:29:30.497698 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.496162 5998 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 01:29:30.497698 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.496171 5998 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 01:29:30.497698 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.496223 5998 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 01:29:30.497698 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.496231 5998 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 01:29:30.497698 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.496299 5998 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 01:29:30.497698 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.496208 5998 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 01:23:21 +0000 UTC" deadline="2028-04-01 18:11:13.70370258 +0000 UTC" Oct 30 01:29:30.497698 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.496341 5998 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12472h41m43.207365986s" Oct 30 01:29:30.492621 ip-10-0-130-0 systemd[1]: Started Kubernetes Kubelet. Oct 30 01:29:30.500228 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:29:30.500181 5998 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-0.ec2.internal\" not found" Oct 30 01:29:30.505224 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505182 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1facbe7-deb0-4fec-94de-42a0dd91ff4f" volumeName="kubernetes.io/empty-dir/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-tmp" seLinuxMountContext="" Oct 30 01:29:30.505314 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505225 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d50883a3-5056-4826-884e-90fe75441e5d" volumeName="kubernetes.io/projected/d50883a3-5056-4826-884e-90fe75441e5d-kube-api-access-gwl72" seLinuxMountContext="" Oct 30 01:29:30.505314 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505240 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0bbf87d-7baa-4b99-9b80-695c2c99c4c8" volumeName="kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret" seLinuxMountContext="" Oct 30 01:29:30.505314 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505253 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 01:29:30.505314 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505263 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da638295-de7f-462a-9a14-2b8105e636b6" volumeName="kubernetes.io/empty-dir/da638295-de7f-462a-9a14-2b8105e636b6-tmp-dir" seLinuxMountContext="" Oct 30 01:29:30.505314 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505274 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da638295-de7f-462a-9a14-2b8105e636b6" volumeName="kubernetes.io/projected/da638295-de7f-462a-9a14-2b8105e636b6-kube-api-access-scl9k" seLinuxMountContext="" Oct 30 01:29:30.505314 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505288 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43e8ec8c-149e-4257-a891-7b117a79c319" volumeName="kubernetes.io/secret/43e8ec8c-149e-4257-a891-7b117a79c319-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 01:29:30.505314 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505299 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fa1c18e-5933-4e09-85f8-28033e6cc4df" volumeName="kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 01:29:30.505314 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505314 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4c33058-34c5-49f3-b803-984c54ff5dad" volumeName="kubernetes.io/projected/b4c33058-34c5-49f3-b803-984c54ff5dad-kube-api-access-zqpgm" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505325 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1facbe7-deb0-4fec-94de-42a0dd91ff4f" volumeName="kubernetes.io/empty-dir/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-tuned" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505337 5998 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505337 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07c8420b-05bf-4d3e-a1a2-7a6d55fffad4" volumeName="kubernetes.io/configmap/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4-iptables-alerter-script" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505358 5998 factory.go:55] Registering systemd factory Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505363 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40d2d25c-e7a0-4ded-9674-19c6d371522e" volumeName="kubernetes.io/configmap/40d2d25c-e7a0-4ded-9674-19c6d371522e-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505377 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fa1c18e-5933-4e09-85f8-28033e6cc4df" volumeName="kubernetes.io/configmap/4fa1c18e-5933-4e09-85f8-28033e6cc4df-metrics-client-ca" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505389 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505403 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505415 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d9a754-4e82-4dd5-b423-50624589bdda" volumeName="kubernetes.io/configmap/a0d9a754-4e82-4dd5-b423-50624589bdda-konnectivity-ca" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505427 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4c33058-34c5-49f3-b803-984c54ff5dad" volumeName="kubernetes.io/secret/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505440 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c01f6b4a-5666-4734-b403-3818692c3a74" volumeName="kubernetes.io/configmap/c01f6b4a-5666-4734-b403-3818692c3a74-ovnkube-script-lib" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505453 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="174be2db-9c13-45fe-b91f-c3302ecb57e2" volumeName="kubernetes.io/empty-dir/174be2db-9c13-45fe-b91f-c3302ecb57e2-tmp-dir" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505464 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3657b83a-1cdf-44de-9a9d-2800cfae5524" volumeName="kubernetes.io/configmap/3657b83a-1cdf-44de-9a9d-2800cfae5524-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505476 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3657b83a-1cdf-44de-9a9d-2800cfae5524" volumeName="kubernetes.io/secret/3657b83a-1cdf-44de-9a9d-2800cfae5524-client-ca-bundle" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505487 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40d2d25c-e7a0-4ded-9674-19c6d371522e" volumeName="kubernetes.io/configmap/40d2d25c-e7a0-4ded-9674-19c6d371522e-cni-binary-copy" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505499 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fa1c18e-5933-4e09-85f8-28033e6cc4df" volumeName="kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505511 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4c33058-34c5-49f3-b803-984c54ff5dad" volumeName="kubernetes.io/secret/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-tls" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505525 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da638295-de7f-462a-9a14-2b8105e636b6" volumeName="kubernetes.io/configmap/da638295-de7f-462a-9a14-2b8105e636b6-config-volume" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505537 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43e8ec8c-149e-4257-a891-7b117a79c319" volumeName="kubernetes.io/projected/43e8ec8c-149e-4257-a891-7b117a79c319-kube-api-access-ssfgz" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505549 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505562 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505578 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c01f6b4a-5666-4734-b403-3818692c3a74" volumeName="kubernetes.io/configmap/c01f6b4a-5666-4734-b403-3818692c3a74-env-overrides" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505590 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da638295-de7f-462a-9a14-2b8105e636b6" volumeName="kubernetes.io/secret/da638295-de7f-462a-9a14-2b8105e636b6-metrics-tls" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505604 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fa1c18e-5933-4e09-85f8-28033e6cc4df" volumeName="kubernetes.io/empty-dir/4fa1c18e-5933-4e09-85f8-28033e6cc4df-tmp" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505617 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-db" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505628 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c01f6b4a-5666-4734-b403-3818692c3a74" volumeName="kubernetes.io/projected/c01f6b4a-5666-4734-b403-3818692c3a74-kube-api-access-kkjlt" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505640 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cee84e-69dd-4296-98d7-de2e6e0c6fe0" volumeName="kubernetes.io/projected/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-kube-api-access-dbxj6" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505652 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ff44b73-9f64-4fca-95f8-6a5a24540843" volumeName="kubernetes.io/projected/6ff44b73-9f64-4fca-95f8-6a5a24540843-kube-api-access-kwgtk" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505665 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505677 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505689 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-tls-assets" seLinuxMountContext="" Oct 30 01:29:30.505704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505701 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-kube-api-access-wspxg" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505747 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91d108c9-4e8c-4d00-8e42-1eecd52741a2" volumeName="kubernetes.io/projected/91d108c9-4e8c-4d00-8e42-1eecd52741a2-kube-api-access-2hwlz" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505761 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4c33058-34c5-49f3-b803-984c54ff5dad" volumeName="kubernetes.io/empty-dir/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-textfile" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505367 5998 factory.go:223] Registration of the systemd container factory successfully Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505773 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43e8ec8c-149e-4257-a891-7b117a79c319" volumeName="kubernetes.io/secret/43e8ec8c-149e-4257-a891-7b117a79c319-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505787 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f486e9e-aa32-4a9a-877b-eca8dbba7091" volumeName="kubernetes.io/projected/4f486e9e-aa32-4a9a-877b-eca8dbba7091-kube-api-access-wvtjv" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505799 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bde30efc-28f6-4fb1-83b7-2b59d6d34359" volumeName="kubernetes.io/projected/bde30efc-28f6-4fb1-83b7-2b59d6d34359-kube-api-access-wd6sl" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505810 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c01f6b4a-5666-4734-b403-3818692c3a74" volumeName="kubernetes.io/configmap/c01f6b4a-5666-4734-b403-3818692c3a74-ovnkube-config" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505819 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1facbe7-deb0-4fec-94de-42a0dd91ff4f" volumeName="kubernetes.io/projected/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-kube-api-access-vtwhw" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505830 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fa1c18e-5933-4e09-85f8-28033e6cc4df" volumeName="kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505841 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="174be2db-9c13-45fe-b91f-c3302ecb57e2" volumeName="kubernetes.io/projected/174be2db-9c13-45fe-b91f-c3302ecb57e2-kube-api-access-jjjjr" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505853 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43e8ec8c-149e-4257-a891-7b117a79c319" volumeName="kubernetes.io/configmap/43e8ec8c-149e-4257-a891-7b117a79c319-metrics-client-ca" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505867 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505877 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505889 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91d108c9-4e8c-4d00-8e42-1eecd52741a2" volumeName="kubernetes.io/secret/91d108c9-4e8c-4d00-8e42-1eecd52741a2-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505900 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3657b83a-1cdf-44de-9a9d-2800cfae5524" volumeName="kubernetes.io/secret/3657b83a-1cdf-44de-9a9d-2800cfae5524-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505910 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505922 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c01f6b4a-5666-4734-b403-3818692c3a74" volumeName="kubernetes.io/secret/c01f6b4a-5666-4734-b403-3818692c3a74-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505932 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cee84e-69dd-4296-98d7-de2e6e0c6fe0" volumeName="kubernetes.io/configmap/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-daemon-config" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505956 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3657b83a-1cdf-44de-9a9d-2800cfae5524" volumeName="kubernetes.io/projected/3657b83a-1cdf-44de-9a9d-2800cfae5524-kube-api-access-xtbks" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505987 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40d2d25c-e7a0-4ded-9674-19c6d371522e" volumeName="kubernetes.io/projected/40d2d25c-e7a0-4ded-9674-19c6d371522e-kube-api-access-qwzmn" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.505997 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fa1c18e-5933-4e09-85f8-28033e6cc4df" volumeName="kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506004 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506013 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40d2d25c-e7a0-4ded-9674-19c6d371522e" volumeName="kubernetes.io/configmap/40d2d25c-e7a0-4ded-9674-19c6d371522e-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506021 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fa1c18e-5933-4e09-85f8-28033e6cc4df" volumeName="kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-grpc-tls" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506028 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506042 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91d108c9-4e8c-4d00-8e42-1eecd52741a2" volumeName="kubernetes.io/empty-dir/91d108c9-4e8c-4d00-8e42-1eecd52741a2-data-volume" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506050 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07c8420b-05bf-4d3e-a1a2-7a6d55fffad4" volumeName="kubernetes.io/projected/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4-kube-api-access-qgnfn" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506056 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3657b83a-1cdf-44de-9a9d-2800cfae5524" volumeName="kubernetes.io/configmap/3657b83a-1cdf-44de-9a9d-2800cfae5524-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506064 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3657b83a-1cdf-44de-9a9d-2800cfae5524" volumeName="kubernetes.io/secret/3657b83a-1cdf-44de-9a9d-2800cfae5524-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506072 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fa1c18e-5933-4e09-85f8-28033e6cc4df" volumeName="kubernetes.io/projected/4fa1c18e-5933-4e09-85f8-28033e6cc4df-kube-api-access-n55vv" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506080 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fa1c18e-5933-4e09-85f8-28033e6cc4df" volumeName="kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506103 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506115 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506125 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0d9a754-4e82-4dd5-b423-50624589bdda" volumeName="kubernetes.io/secret/a0d9a754-4e82-4dd5-b423-50624589bdda-agent-certs" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506133 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18cee84e-69dd-4296-98d7-de2e6e0c6fe0" volumeName="kubernetes.io/configmap/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-cni-binary-copy" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506141 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3657b83a-1cdf-44de-9a9d-2800cfae5524" volumeName="kubernetes.io/empty-dir/3657b83a-1cdf-44de-9a9d-2800cfae5524-audit-log" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506149 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f486e9e-aa32-4a9a-877b-eca8dbba7091" volumeName="kubernetes.io/configmap/4f486e9e-aa32-4a9a-877b-eca8dbba7091-serviceca" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506156 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ff44b73-9f64-4fca-95f8-6a5a24540843" volumeName="kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506168 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8af570c7-62ff-4f33-93df-1714b5ffc20c" volumeName="kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506176 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-web-config" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506184 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4c33058-34c5-49f3-b803-984c54ff5dad" volumeName="kubernetes.io/configmap/b4c33058-34c5-49f3-b803-984c54ff5dad-metrics-client-ca" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506193 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bde30efc-28f6-4fb1-83b7-2b59d6d34359" volumeName="kubernetes.io/secret/bde30efc-28f6-4fb1-83b7-2b59d6d34359-cert" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506201 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config-out" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506210 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" volumeName="kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-grpc-tls" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506217 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91d108c9-4e8c-4d00-8e42-1eecd52741a2" volumeName="kubernetes.io/configmap/91d108c9-4e8c-4d00-8e42-1eecd52741a2-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506225 5998 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4c33058-34c5-49f3-b803-984c54ff5dad" volumeName="kubernetes.io/configmap/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506233 5998 reconstruct.go:97] "Volume reconstruction finished" Oct 30 01:29:30.506652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.506239 5998 reconciler.go:29] "Reconciler: start to sync state" Oct 30 01:29:30.508629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.508236 5998 factory.go:153] Registering CRI-O factory Oct 30 01:29:30.508629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.508256 5998 factory.go:223] Registration of the crio container factory successfully Oct 30 01:29:30.508629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.508278 5998 factory.go:103] Registering Raw factory Oct 30 01:29:30.508629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.508293 5998 manager.go:1199] Started watching for new ooms in manager Oct 30 01:29:30.508629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.508585 5998 manager.go:319] Starting recovery of all containers Oct 30 01:29:30.530181 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.529381 5998 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 01:29:30.530885 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.530861 5998 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 01:29:30.530885 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.530886 5998 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 01:29:30.531056 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.530909 5998 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 01:29:30.531937 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:29:30.531709 5998 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 01:29:30.552490 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.552422 5998 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 01:29:30.559369 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.559340 5998 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:29:30.559497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.559364 5998 generic.go:356] "Generic (PLEG): container finished" podID="40d2d25c-e7a0-4ded-9674-19c6d371522e" containerID="ae0cf2c9763b957c33dc1285221b7d6b1b049258837608673d9044aa66f37c76" exitCode=0 Oct 30 01:29:30.559497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.559394 5998 generic.go:356] "Generic (PLEG): container finished" podID="40d2d25c-e7a0-4ded-9674-19c6d371522e" containerID="23aefc04aa7988b03d3221c52dddd202e3592ec6c1666347ec04b658596083c6" exitCode=0 Oct 30 01:29:30.559497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.559405 5998 generic.go:356] "Generic (PLEG): container finished" podID="40d2d25c-e7a0-4ded-9674-19c6d371522e" containerID="8d0cd7fbfdf242a87ecff4368ad98d35d18422495cf3c8b10ee2a881dc8636f1" exitCode=0 Oct 30 01:29:30.559497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.559413 5998 generic.go:356] "Generic (PLEG): container finished" podID="40d2d25c-e7a0-4ded-9674-19c6d371522e" containerID="82a3e1615c34f568c7d52b6fad0d6f2ca4bd181b4bf7affe9af532f25d4c2e95" exitCode=0 Oct 30 01:29:30.559497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.559420 5998 generic.go:356] "Generic (PLEG): container finished" podID="40d2d25c-e7a0-4ded-9674-19c6d371522e" containerID="60392adfc79717a9d92b7d8dcd47acd39253f2eba6f406c922d28ae707d5c4f6" exitCode=0 Oct 30 01:29:30.559497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.559428 5998 generic.go:356] "Generic (PLEG): container finished" podID="40d2d25c-e7a0-4ded-9674-19c6d371522e" containerID="9672f4742ddd3de805c65ac08e69025fc12592a746f02c5d00785aa1e7f9d963" exitCode=0 Oct 30 01:29:30.562152 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.562133 5998 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:29:30.569581 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.569553 5998 generic.go:356] "Generic (PLEG): container finished" podID="f54d525f1aa0b0582cbd13c31e87b9a6" containerID="e9b9d416301b94032e01caf0dc6ea22884019e88a4b2de8af8df180527542e45" exitCode=0 Oct 30 01:29:30.573875 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.573855 5998 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:29:30.575304 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.575287 5998 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:29:30.575435 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.575407 5998 generic.go:356] "Generic (PLEG): container finished" podID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerID="a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e" exitCode=0 Oct 30 01:29:30.580210 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.580182 5998 generic.go:356] "Generic (PLEG): container finished" podID="b4c33058-34c5-49f3-b803-984c54ff5dad" containerID="e3d9205c4b9f3b7afa6e7dd89b8290b46407ad48099b5fe14fca6c98e44805be" exitCode=0 Oct 30 01:29:30.611303 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.611277 5998 manager.go:324] Recovery completed Oct 30 01:29:30.632137 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.632115 5998 cpu_manager.go:226] "Starting" policy="none" Oct 30 01:29:30.632137 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.632130 5998 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 01:29:30.632137 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:29:30.632123 5998 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 01:29:30.632334 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.632148 5998 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 01:29:30.632334 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.632281 5998 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 01:29:30.632334 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.632291 5998 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 01:29:30.632452 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.632441 5998 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 01:29:30.632452 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.632452 5998 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 01:29:30.632508 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.632459 5998 policy_none.go:50] "Start" Oct 30 01:29:30.632508 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.632469 5998 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 01:29:30.632508 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.632479 5998 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 01:29:30.633542 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.633529 5998 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 01:29:30.633577 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.633542 5998 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 01:29:30.633577 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.633549 5998 policy_none.go:44] "Start" Oct 30 01:29:30.638260 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.638228 5998 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 01:29:30.640813 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.640799 5998 manager.go:341] "Starting Device Plugin manager" Oct 30 01:29:30.641522 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:29:30.640828 5998 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 01:29:30.641522 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.641524 5998 server.go:87] "Starting device plugin registration server" Oct 30 01:29:30.641827 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.641812 5998 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 01:29:30.641889 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.641826 5998 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 01:29:30.641942 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.641919 5998 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 01:29:30.642076 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.642041 5998 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 01:29:30.642165 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.642082 5998 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 01:29:30.642165 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.642100 5998 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 01:29:30.642261 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.642145 5998 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T01:29:30.642073805Z","UUID":"2db31162-b289-4814-8fbd-79c89cc23691","Handler":null,"Name":"","Endpoint":""} Oct 30 01:29:30.643762 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:29:30.643739 5998 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 01:29:30.644203 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.644181 5998 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 01:29:30.644281 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.644233 5998 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 01:29:30.742204 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.742133 5998 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:29:30.744210 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.744188 5998 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:29:30.744342 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.744221 5998 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:29:30.744342 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.744232 5998 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-0.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:29:30.744342 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.744271 5998 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-0.ec2.internal" Oct 30 01:29:30.777480 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.777455 5998 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-0.ec2.internal" Oct 30 01:29:30.777591 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.777531 5998 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-0.ec2.internal" Oct 30 01:29:30.833318 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.833249 5998 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal"] Oct 30 01:29:30.833814 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.833781 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal" event={"ID":"8da95128540617dd483ff23c5d0ddd95","Type":"ContainerStarted","Data":"92bc8c9e67f63521f31b6ee495a6da4a2cfab35ce71e9e29364344f4a272ca7f"} Oct 30 01:29:30.833876 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.833820 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal" event={"ID":"8da95128540617dd483ff23c5d0ddd95","Type":"ContainerStarted","Data":"1d7fdd368334d11a5e39d01f71bdac4b4c5de1013b365826129fe7082e083759"} Oct 30 01:29:30.833926 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.833887 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" event={"ID":"f54d525f1aa0b0582cbd13c31e87b9a6","Type":"ContainerStarted","Data":"77dcb2287173250bf1a38b22c94c832988db660739035c57c87554bb54e194aa"} Oct 30 01:29:30.833926 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.833903 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" event={"ID":"f54d525f1aa0b0582cbd13c31e87b9a6","Type":"ContainerDied","Data":"e9b9d416301b94032e01caf0dc6ea22884019e88a4b2de8af8df180527542e45"} Oct 30 01:29:30.833926 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.833919 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" event={"ID":"f54d525f1aa0b0582cbd13c31e87b9a6","Type":"ContainerStarted","Data":"c4320bb0b198f2a9176fd4a26532dc30e402064783fc6444fd2f4efc57ab46bc"} Oct 30 01:29:30.834062 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.833888 5998 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:29:30.834133 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.834067 5998 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal" Oct 30 01:29:30.871446 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.871410 5998 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:29:30.871583 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:29:30.871468 5998 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-0.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal" Oct 30 01:29:30.872036 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.872020 5998 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:29:30.872156 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:29:30.872072 5998 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:29:30.955370 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.955320 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f54d525f1aa0b0582cbd13c31e87b9a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal\" (UID: \"f54d525f1aa0b0582cbd13c31e87b9a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:29:30.955370 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.955374 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f54d525f1aa0b0582cbd13c31e87b9a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal\" (UID: \"f54d525f1aa0b0582cbd13c31e87b9a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:29:30.955572 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:30.955422 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8da95128540617dd483ff23c5d0ddd95-config\") pod \"kube-apiserver-proxy-ip-10-0-130-0.ec2.internal\" (UID: \"8da95128540617dd483ff23c5d0ddd95\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal" Oct 30 01:29:31.056030 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.055928 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8da95128540617dd483ff23c5d0ddd95-config\") pod \"kube-apiserver-proxy-ip-10-0-130-0.ec2.internal\" (UID: \"8da95128540617dd483ff23c5d0ddd95\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal" Oct 30 01:29:31.056030 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.055975 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f54d525f1aa0b0582cbd13c31e87b9a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal\" (UID: \"f54d525f1aa0b0582cbd13c31e87b9a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:29:31.056030 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.055997 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f54d525f1aa0b0582cbd13c31e87b9a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal\" (UID: \"f54d525f1aa0b0582cbd13c31e87b9a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:29:31.056289 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.056052 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f54d525f1aa0b0582cbd13c31e87b9a6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal\" (UID: \"f54d525f1aa0b0582cbd13c31e87b9a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:29:31.056289 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.056052 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f54d525f1aa0b0582cbd13c31e87b9a6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal\" (UID: \"f54d525f1aa0b0582cbd13c31e87b9a6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:29:31.056289 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.056079 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8da95128540617dd483ff23c5d0ddd95-config\") pod \"kube-apiserver-proxy-ip-10-0-130-0.ec2.internal\" (UID: \"8da95128540617dd483ff23c5d0ddd95\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal" Oct 30 01:29:31.484658 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.484622 5998 apiserver.go:52] "Watching apiserver" Oct 30 01:29:31.527450 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.527412 5998 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 01:29:31.529193 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.529161 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-zrv6d","openshift-dns/dns-default-6vkc5","openshift-ingress-canary/ingress-canary-tlpbm","openshift-monitoring/prometheus-k8s-0","openshift-insights/insights-runtime-extractor-vccw5","kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal","openshift-network-operator/iptables-alerter-p9b7v","openshift-image-registry/node-ca-lq7lj","openshift-monitoring/metrics-server-868b7dc86d-hzrmz","openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq","kube-system/konnectivity-agent-44ftt","openshift-ovn-kubernetes/ovnkube-node-4qqkx","openshift-multus/network-metrics-daemon-plsqk","kube-system/global-pull-secret-syncer-55kfb","openshift-cluster-node-tuning-operator/tuned-kjlnq","openshift-network-diagnostics/network-check-target-pct74","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s","openshift-dns/node-resolver-8wd8v","openshift-monitoring/node-exporter-nd754","openshift-multus/multus-sgm2w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal","openshift-monitoring/thanos-querier-767946cb5-q64ns"] Oct 30 01:29:31.544254 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.544229 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-lkdgr\"" Oct 30 01:29:31.544254 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.544229 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hp2kr\"" Oct 30 01:29:31.544694 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.544678 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 01:29:31.544808 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.544691 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 01:29:31.544927 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.544808 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 01:29:31.544927 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.544814 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 01:29:31.545034 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.545007 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 01:29:31.545034 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.545012 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 01:29:31.545127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.545044 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 01:29:31.545426 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.545413 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 01:29:31.545521 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.545508 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 01:29:31.545567 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.545547 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 01:29:31.545757 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.545741 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 01:29:31.545837 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.545822 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 01:29:31.545873 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.545830 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 01:29:31.545912 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.545871 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 01:29:31.545948 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.545911 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 01:29:31.546239 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.546224 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 01:29:31.546357 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.546341 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 01:29:31.546418 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.546346 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rwcx9\"" Oct 30 01:29:31.546418 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.546370 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lzbd2\"" Oct 30 01:29:31.546418 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.546398 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lm6wv\"" Oct 30 01:29:31.546418 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.546411 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2gfdm\"" Oct 30 01:29:31.551352 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.551340 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 01:29:31.552978 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.552965 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 01:29:31.553721 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.553706 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j7lll\"" Oct 30 01:29:31.553871 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.553859 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 01:29:31.553913 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.553900 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sfxzn\"" Oct 30 01:29:31.556962 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.556707 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:29:31.557158 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.557137 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:29:31.557638 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.557615 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2p848\"" Oct 30 01:29:31.558228 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.558205 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 01:29:31.558298 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.558272 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 01:29:31.558345 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.558327 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 01:29:31.559307 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.559272 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 01:29:31.559498 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.559484 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 01:29:31.559663 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.559649 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 01:29:31.559723 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.559665 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tntm7\"" Oct 30 01:29:31.559723 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.559721 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 01:29:31.559810 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.559789 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7p9xn\"" Oct 30 01:29:31.559895 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.559874 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 01:29:31.559942 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.559925 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 01:29:31.560000 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.559980 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c6clm\"" Oct 30 01:29:31.560000 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.559995 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 01:29:31.560104 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.560052 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nc846\"" Oct 30 01:29:31.560104 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.560052 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 01:29:31.560201 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.560153 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xglfc\"" Oct 30 01:29:31.560289 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.560272 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vfcpg\"" Oct 30 01:29:31.560768 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.560751 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 01:29:31.560839 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.560812 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zqs4x\"" Oct 30 01:29:31.561678 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.561648 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 01:29:31.561772 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.561679 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 01:29:31.561772 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.561745 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-lzvbc\"" Oct 30 01:29:31.561927 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.561778 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 01:29:31.561927 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.561833 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 01:29:31.562011 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.561936 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 01:29:31.562011 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.562002 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 01:29:31.562159 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.562147 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 01:29:31.564419 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.564405 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 01:29:31.564488 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.564444 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 01:29:31.564828 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.564805 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 01:29:31.565289 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.565274 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 01:29:31.565361 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.565301 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 01:29:31.565361 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.565309 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 01:29:31.565361 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.565316 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 01:29:31.565361 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.565322 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xk9xs\"" Oct 30 01:29:31.565361 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.565304 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 01:29:31.565361 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.565333 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 01:29:31.565361 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.565308 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 01:29:31.565361 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.565316 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 01:29:31.565361 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.565335 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 01:29:31.565624 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.565384 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 01:29:31.565679 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.565659 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 01:29:31.565782 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.565700 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 01:29:31.565782 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.565727 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 01:29:31.565782 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.565734 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 01:29:31.565893 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.565843 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 01:29:31.565893 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.565848 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 01:29:31.565975 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.565954 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 01:29:31.566059 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.566047 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 01:29:31.566127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.566078 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 01:29:31.566205 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.566193 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-569c19rarmjmk\"" Oct 30 01:29:31.566251 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.566206 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 01:29:31.566251 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.566209 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 01:29:31.566251 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.566236 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 01:29:31.566405 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.566262 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3a819liq0v26f\"" Oct 30 01:29:31.566405 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.566270 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 01:29:31.566405 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.566264 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nw28w\"" Oct 30 01:29:31.566405 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.566350 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rgh9c\"" Oct 30 01:29:31.566405 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.566384 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 01:29:31.566566 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.566421 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 01:29:31.568484 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.568470 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-435e5nstmdbes\"" Oct 30 01:29:31.573770 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.573746 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 01:29:31.573906 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.573892 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 01:29:31.581725 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.581703 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:31.592719 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.592703 5998 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal" Oct 30 01:29:31.592922 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.592907 5998 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:29:31.602810 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.602789 5998 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 01:29:31.614878 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.614858 5998 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:29:31.614991 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:29:31.614907 5998 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-0.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-0.ec2.internal" Oct 30 01:29:31.615341 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.615326 5998 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:29:31.615412 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:29:31.615360 5998 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-0.ec2.internal" Oct 30 01:29:31.658358 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658331 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3657b83a-1cdf-44de-9a9d-2800cfae5524-metrics-server-audit-profiles\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:31.658358 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658358 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da638295-de7f-462a-9a14-2b8105e636b6-metrics-tls\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:31.658543 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658383 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.658543 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658401 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.658543 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658419 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.658543 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658446 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-device-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.658543 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658472 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43e8ec8c-149e-4257-a891-7b117a79c319-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:31.658543 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658532 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.658852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658576 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3657b83a-1cdf-44de-9a9d-2800cfae5524-secret-metrics-server-tls\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:31.658852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658606 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwzmn\" (UniqueName: \"kubernetes.io/projected/40d2d25c-e7a0-4ded-9674-19c6d371522e-kube-api-access-qwzmn\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:29:31.658852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658636 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bde30efc-28f6-4fb1-83b7-2b59d6d34359-cert\") pod \"ingress-canary-tlpbm\" (UID: \"bde30efc-28f6-4fb1-83b7-2b59d6d34359\") " pod="openshift-ingress-canary/ingress-canary-tlpbm" Oct 30 01:29:31.658852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658666 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fa1c18e-5933-4e09-85f8-28033e6cc4df-metrics-client-ca\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:31.658852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658671 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.658852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658665 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/da638295-de7f-462a-9a14-2b8105e636b6-metrics-tls\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:31.658852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658716 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3657b83a-1cdf-44de-9a9d-2800cfae5524-metrics-server-audit-profiles\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:31.658852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658739 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.658852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658747 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43e8ec8c-149e-4257-a891-7b117a79c319-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:31.658852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658767 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-tmp\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.658852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658825 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c01f6b4a-5666-4734-b403-3818692c3a74-ovn-node-metrics-cert\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.658852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658851 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3657b83a-1cdf-44de-9a9d-2800cfae5524-audit-log\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658877 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scl9k\" (UniqueName: \"kubernetes.io/projected/da638295-de7f-462a-9a14-2b8105e636b6-kube-api-access-scl9k\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658882 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658904 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-textfile\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658934 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-grpc-tls\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658945 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bde30efc-28f6-4fb1-83b7-2b59d6d34359-cert\") pod \"ingress-canary-tlpbm\" (UID: \"bde30efc-28f6-4fb1-83b7-2b59d6d34359\") " pod="openshift-ingress-canary/ingress-canary-tlpbm" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658890 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3657b83a-1cdf-44de-9a9d-2800cfae5524-secret-metrics-server-tls\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658972 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-tmp\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658932 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fa1c18e-5933-4e09-85f8-28033e6cc4df-metrics-client-ca\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658982 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3657b83a-1cdf-44de-9a9d-2800cfae5524-audit-log\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658961 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-sys-fs\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659001 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-textfile\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.658999 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659112 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkjlt\" (UniqueName: \"kubernetes.io/projected/c01f6b4a-5666-4734-b403-3818692c3a74-kube-api-access-kkjlt\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659150 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c01f6b4a-5666-4734-b403-3818692c3a74-ovn-node-metrics-cert\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659153 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40d2d25c-e7a0-4ded-9674-19c6d371522e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659200 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b4c33058-34c5-49f3-b803-984c54ff5dad-root\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659225 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659250 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-run-multus-certs\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659302 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-grpc-tls\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659316 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659332 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/40d2d25c-e7a0-4ded-9674-19c6d371522e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:29:31.659378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659376 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-cnibin\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659473 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40d2d25c-e7a0-4ded-9674-19c6d371522e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659507 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-sysctl-conf\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659512 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659540 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659660 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/40d2d25c-e7a0-4ded-9674-19c6d371522e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659672 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659692 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-kubelet\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659708 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-run-openvswitch\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659725 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3657b83a-1cdf-44de-9a9d-2800cfae5524-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659741 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-socket-dir-parent\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659756 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-run-k8s-cni-cncf-io\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659783 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659786 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0d9a754-4e82-4dd5-b423-50624589bdda-agent-certs\") pod \"konnectivity-agent-44ftt\" (UID: \"a0d9a754-4e82-4dd5-b423-50624589bdda\") " pod="kube-system/konnectivity-agent-44ftt" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659831 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659933 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659948 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-run-ovn\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659976 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-system-cni-dir\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.659999 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0d9a754-4e82-4dd5-b423-50624589bdda-agent-certs\") pod \"konnectivity-agent-44ftt\" (UID: \"a0d9a754-4e82-4dd5-b423-50624589bdda\") " pod="kube-system/konnectivity-agent-44ftt" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660008 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-etc-kubernetes\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660008 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3657b83a-1cdf-44de-9a9d-2800cfae5524-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660034 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/91d108c9-4e8c-4d00-8e42-1eecd52741a2-crio-socket\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660035 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660064 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f486e9e-aa32-4a9a-877b-eca8dbba7091-serviceca\") pod \"node-ca-lq7lj\" (UID: \"4f486e9e-aa32-4a9a-877b-eca8dbba7091\") " pod="openshift-image-registry/node-ca-lq7lj" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660082 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-run-netns\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660123 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-run-netns\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660151 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660168 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40d2d25c-e7a0-4ded-9674-19c6d371522e-system-cni-dir\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660211 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4-host-slash\") pod \"iptables-alerter-p9b7v\" (UID: \"07c8420b-05bf-4d3e-a1a2-7a6d55fffad4\") " pod="openshift-network-operator/iptables-alerter-p9b7v" Oct 30 01:29:31.660255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660228 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-run-systemd\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660270 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-sysconfig\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660309 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs\") pod \"network-metrics-daemon-plsqk\" (UID: \"6ff44b73-9f64-4fca-95f8-6a5a24540843\") " pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660326 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4-iptables-alerter-script\") pod \"iptables-alerter-p9b7v\" (UID: \"07c8420b-05bf-4d3e-a1a2-7a6d55fffad4\") " pod="openshift-network-operator/iptables-alerter-p9b7v" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660374 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660456 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-lib-modules\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660474 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4-iptables-alerter-script\") pod \"iptables-alerter-p9b7v\" (UID: \"07c8420b-05bf-4d3e-a1a2-7a6d55fffad4\") " pod="openshift-network-operator/iptables-alerter-p9b7v" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660556 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/91d108c9-4e8c-4d00-8e42-1eecd52741a2-data-volume\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660584 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c01f6b4a-5666-4734-b403-3818692c3a74-ovnkube-config\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660619 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff44b73-9f64-4fca-95f8-6a5a24540843-metrics-certs\") pod \"network-metrics-daemon-plsqk\" (UID: \"6ff44b73-9f64-4fca-95f8-6a5a24540843\") " pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660634 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f486e9e-aa32-4a9a-877b-eca8dbba7091-serviceca\") pod \"node-ca-lq7lj\" (UID: \"4f486e9e-aa32-4a9a-877b-eca8dbba7091\") " pod="openshift-image-registry/node-ca-lq7lj" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660638 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjjjr\" (UniqueName: \"kubernetes.io/projected/174be2db-9c13-45fe-b91f-c3302ecb57e2-kube-api-access-jjjjr\") pod \"node-resolver-8wd8v\" (UID: \"174be2db-9c13-45fe-b91f-c3302ecb57e2\") " pod="openshift-dns/node-resolver-8wd8v" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660678 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da638295-de7f-462a-9a14-2b8105e636b6-tmp-dir\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660692 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/91d108c9-4e8c-4d00-8e42-1eecd52741a2-data-volume\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660774 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c01f6b4a-5666-4734-b403-3818692c3a74-ovnkube-config\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660783 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660837 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da638295-de7f-462a-9a14-2b8105e636b6-tmp-dir\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660866 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0d9a754-4e82-4dd5-b423-50624589bdda-konnectivity-ca\") pod \"konnectivity-agent-44ftt\" (UID: \"a0d9a754-4e82-4dd5-b423-50624589bdda\") " pod="kube-system/konnectivity-agent-44ftt" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660937 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91d108c9-4e8c-4d00-8e42-1eecd52741a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.660974 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661023 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n55vv\" (UniqueName: \"kubernetes.io/projected/4fa1c18e-5933-4e09-85f8-28033e6cc4df-kube-api-access-n55vv\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661067 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-log-socket\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661084 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3657b83a-1cdf-44de-9a9d-2800cfae5524-client-ca-bundle\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661126 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0d9a754-4e82-4dd5-b423-50624589bdda-konnectivity-ca\") pod \"konnectivity-agent-44ftt\" (UID: \"a0d9a754-4e82-4dd5-b423-50624589bdda\") " pod="kube-system/konnectivity-agent-44ftt" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661141 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fa1c18e-5933-4e09-85f8-28033e6cc4df-tmp\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661147 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91d108c9-4e8c-4d00-8e42-1eecd52741a2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661190 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fa1c18e-5933-4e09-85f8-28033e6cc4df-tmp\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661193 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661220 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661238 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwgtk\" (UniqueName: \"kubernetes.io/projected/6ff44b73-9f64-4fca-95f8-6a5a24540843-kube-api-access-kwgtk\") pod \"network-metrics-daemon-plsqk\" (UID: \"6ff44b73-9f64-4fca-95f8-6a5a24540843\") " pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661254 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.661333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661323 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3657b83a-1cdf-44de-9a9d-2800cfae5524-client-ca-bundle\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661356 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-run\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661368 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661387 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661370 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661449 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43e8ec8c-149e-4257-a891-7b117a79c319-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661525 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-slash\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661560 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwl72\" (UniqueName: \"kubernetes.io/projected/d50883a3-5056-4826-884e-90fe75441e5d-kube-api-access-gwl72\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661587 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssfgz\" (UniqueName: \"kubernetes.io/projected/43e8ec8c-149e-4257-a891-7b117a79c319-kube-api-access-ssfgz\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661608 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c01f6b4a-5666-4734-b403-3818692c3a74-env-overrides\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661621 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/43e8ec8c-149e-4257-a891-7b117a79c319-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661634 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-cni-dir\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661657 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-host\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661739 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-original-pull-secret\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661744 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c01f6b4a-5666-4734-b403-3818692c3a74-env-overrides\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661762 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtwhw\" (UniqueName: \"kubernetes.io/projected/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-kube-api-access-vtwhw\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661796 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f486e9e-aa32-4a9a-877b-eca8dbba7091-host\") pod \"node-ca-lq7lj\" (UID: \"4f486e9e-aa32-4a9a-877b-eca8dbba7091\") " pod="openshift-image-registry/node-ca-lq7lj" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661820 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40d2d25c-e7a0-4ded-9674-19c6d371522e-cnibin\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661872 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-tls\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661904 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-var-lib-kubelet\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661930 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661954 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbxj6\" (UniqueName: \"kubernetes.io/projected/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-kube-api-access-dbxj6\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.661985 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40d2d25c-e7a0-4ded-9674-19c6d371522e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662084 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-modprobe-d\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662122 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hwlz\" (UniqueName: \"kubernetes.io/projected/91d108c9-4e8c-4d00-8e42-1eecd52741a2-kube-api-access-2hwlz\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662162 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-tls\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662161 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-tls\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662191 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662198 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662244 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662332 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-etc-openvswitch\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662375 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-socket-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662382 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662419 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-tls\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662440 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-run-ovn-kubernetes\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662450 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662461 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662466 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4c33058-34c5-49f3-b803-984c54ff5dad-sys\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662510 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wspxg\" (UniqueName: \"kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-kube-api-access-wspxg\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662539 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662562 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-dbus\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662660 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd6sl\" (UniqueName: \"kubernetes.io/projected/bde30efc-28f6-4fb1-83b7-2b59d6d34359-kube-api-access-wd6sl\") pod \"ingress-canary-tlpbm\" (UID: \"bde30efc-28f6-4fb1-83b7-2b59d6d34359\") " pod="openshift-ingress-canary/ingress-canary-tlpbm" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662686 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-kubernetes\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662713 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-tuned\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662739 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91d108c9-4e8c-4d00-8e42-1eecd52741a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662764 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvtjv\" (UniqueName: \"kubernetes.io/projected/4f486e9e-aa32-4a9a-877b-eca8dbba7091-kube-api-access-wvtjv\") pod \"node-ca-lq7lj\" (UID: \"4f486e9e-aa32-4a9a-877b-eca8dbba7091\") " pod="openshift-image-registry/node-ca-lq7lj" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662792 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtbks\" (UniqueName: \"kubernetes.io/projected/3657b83a-1cdf-44de-9a9d-2800cfae5524-kube-api-access-xtbks\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662814 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-var-lib-kubelet\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662828 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-hostroot\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662846 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgnfn\" (UniqueName: \"kubernetes.io/projected/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4-kube-api-access-qgnfn\") pod \"iptables-alerter-p9b7v\" (UID: \"07c8420b-05bf-4d3e-a1a2-7a6d55fffad4\") " pod="openshift-network-operator/iptables-alerter-p9b7v" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662863 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662901 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-tuned\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662921 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-cni-bin\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662944 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91d108c9-4e8c-4d00-8e42-1eecd52741a2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662952 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-cni-binary-copy\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.662977 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-var-lib-cni-bin\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.663009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663010 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-sysctl-d\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663052 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/174be2db-9c13-45fe-b91f-c3302ecb57e2-hosts-file\") pod \"node-resolver-8wd8v\" (UID: \"174be2db-9c13-45fe-b91f-c3302ecb57e2\") " pod="openshift-dns/node-resolver-8wd8v" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663054 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663126 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-conf-dir\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663155 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-cni-binary-copy\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663158 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da638295-de7f-462a-9a14-2b8105e636b6-config-volume\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663192 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4c33058-34c5-49f3-b803-984c54ff5dad-metrics-client-ca\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663209 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-cni-netd\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663238 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663301 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/da638295-de7f-462a-9a14-2b8105e636b6-config-volume\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663581 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663625 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174be2db-9c13-45fe-b91f-c3302ecb57e2-tmp-dir\") pod \"node-resolver-8wd8v\" (UID: \"174be2db-9c13-45fe-b91f-c3302ecb57e2\") " pod="openshift-dns/node-resolver-8wd8v" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663668 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-var-lib-cni-multus\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663694 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4c33058-34c5-49f3-b803-984c54ff5dad-metrics-client-ca\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663703 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqpgm\" (UniqueName: \"kubernetes.io/projected/b4c33058-34c5-49f3-b803-984c54ff5dad-kube-api-access-zqpgm\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663700 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663735 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-sys\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663778 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174be2db-9c13-45fe-b91f-c3302ecb57e2-tmp-dir\") pod \"node-resolver-8wd8v\" (UID: \"174be2db-9c13-45fe-b91f-c3302ecb57e2\") " pod="openshift-dns/node-resolver-8wd8v" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663787 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663809 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663919 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqrp9\" (UniqueName: \"kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9\") pod \"network-check-target-pct74\" (UID: \"8af570c7-62ff-4f33-93df-1714b5ffc20c\") " pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.663961 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43e8ec8c-149e-4257-a891-7b117a79c319-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.664246 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/43e8ec8c-149e-4257-a891-7b117a79c319-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.664260 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-daemon-config\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.664318 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40d2d25c-e7a0-4ded-9674-19c6d371522e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.664353 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.664425 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-accelerators-collector-config\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.664462 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-systemd\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.664469 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.664497 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-registration-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.664684 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-daemon-config\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.664690 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-accelerators-collector-config\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.664729 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.664767 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c01f6b4a-5666-4734-b403-3818692c3a74-ovnkube-script-lib\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.664844 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3657b83a-1cdf-44de-9a9d-2800cfae5524-secret-metrics-server-client-certs\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.664878 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-wtmp\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.664909 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.664934 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.665025 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-var-lib-openvswitch\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.665054 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.665105 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-kubelet-config\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.665136 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-systemd-units\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.665163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.665170 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-node-log\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.666873 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.665551 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.666873 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.665763 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c01f6b4a-5666-4734-b403-3818692c3a74-ovnkube-script-lib\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.666873 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.665828 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.666873 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.666031 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3657b83a-1cdf-44de-9a9d-2800cfae5524-secret-metrics-server-client-certs\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:31.666873 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.666199 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.666873 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.666514 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4fa1c18e-5933-4e09-85f8-28033e6cc4df-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:31.666873 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.666717 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/40d2d25c-e7a0-4ded-9674-19c6d371522e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:29:31.681528 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.681504 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwzmn\" (UniqueName: \"kubernetes.io/projected/40d2d25c-e7a0-4ded-9674-19c6d371522e-kube-api-access-qwzmn\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:29:31.684538 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.684515 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scl9k\" (UniqueName: \"kubernetes.io/projected/da638295-de7f-462a-9a14-2b8105e636b6-kube-api-access-scl9k\") pod \"dns-default-6vkc5\" (UID: \"da638295-de7f-462a-9a14-2b8105e636b6\") " pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:31.684694 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.684674 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkjlt\" (UniqueName: \"kubernetes.io/projected/c01f6b4a-5666-4734-b403-3818692c3a74-kube-api-access-kkjlt\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.687292 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.687276 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtwhw\" (UniqueName: \"kubernetes.io/projected/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-kube-api-access-vtwhw\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.688022 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.688007 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wspxg\" (UniqueName: \"kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-kube-api-access-wspxg\") pod \"prometheus-k8s-0\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:31.689676 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.689661 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjjjr\" (UniqueName: \"kubernetes.io/projected/174be2db-9c13-45fe-b91f-c3302ecb57e2-kube-api-access-jjjjr\") pod \"node-resolver-8wd8v\" (UID: \"174be2db-9c13-45fe-b91f-c3302ecb57e2\") " pod="openshift-dns/node-resolver-8wd8v" Oct 30 01:29:31.690432 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.690417 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hwlz\" (UniqueName: \"kubernetes.io/projected/91d108c9-4e8c-4d00-8e42-1eecd52741a2-kube-api-access-2hwlz\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:31.693363 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.693347 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqpgm\" (UniqueName: \"kubernetes.io/projected/b4c33058-34c5-49f3-b803-984c54ff5dad-kube-api-access-zqpgm\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.696613 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.696594 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbxj6\" (UniqueName: \"kubernetes.io/projected/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-kube-api-access-dbxj6\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.696766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.696746 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwl72\" (UniqueName: \"kubernetes.io/projected/d50883a3-5056-4826-884e-90fe75441e5d-kube-api-access-gwl72\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.697035 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.697019 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssfgz\" (UniqueName: \"kubernetes.io/projected/43e8ec8c-149e-4257-a891-7b117a79c319-kube-api-access-ssfgz\") pod \"openshift-state-metrics-7bd7489957-8dtpq\" (UID: \"43e8ec8c-149e-4257-a891-7b117a79c319\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8dtpq" Oct 30 01:29:31.697035 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.697027 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvtjv\" (UniqueName: \"kubernetes.io/projected/4f486e9e-aa32-4a9a-877b-eca8dbba7091-kube-api-access-wvtjv\") pod \"node-ca-lq7lj\" (UID: \"4f486e9e-aa32-4a9a-877b-eca8dbba7091\") " pod="openshift-image-registry/node-ca-lq7lj" Oct 30 01:29:31.697116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.697031 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgnfn\" (UniqueName: \"kubernetes.io/projected/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4-kube-api-access-qgnfn\") pod \"iptables-alerter-p9b7v\" (UID: \"07c8420b-05bf-4d3e-a1a2-7a6d55fffad4\") " pod="openshift-network-operator/iptables-alerter-p9b7v" Oct 30 01:29:31.697116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.697032 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n55vv\" (UniqueName: \"kubernetes.io/projected/4fa1c18e-5933-4e09-85f8-28033e6cc4df-kube-api-access-n55vv\") pod \"thanos-querier-767946cb5-q64ns\" (UID: \"4fa1c18e-5933-4e09-85f8-28033e6cc4df\") " pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:31.697890 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.697867 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtbks\" (UniqueName: \"kubernetes.io/projected/3657b83a-1cdf-44de-9a9d-2800cfae5524-kube-api-access-xtbks\") pod \"metrics-server-868b7dc86d-hzrmz\" (UID: \"3657b83a-1cdf-44de-9a9d-2800cfae5524\") " pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:31.698293 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.698277 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqrp9\" (UniqueName: \"kubernetes.io/projected/8af570c7-62ff-4f33-93df-1714b5ffc20c-kube-api-access-xqrp9\") pod \"network-check-target-pct74\" (UID: \"8af570c7-62ff-4f33-93df-1714b5ffc20c\") " pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:29:31.699199 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.699184 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd6sl\" (UniqueName: \"kubernetes.io/projected/bde30efc-28f6-4fb1-83b7-2b59d6d34359-kube-api-access-wd6sl\") pod \"ingress-canary-tlpbm\" (UID: \"bde30efc-28f6-4fb1-83b7-2b59d6d34359\") " pod="openshift-ingress-canary/ingress-canary-tlpbm" Oct 30 01:29:31.702985 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.702968 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwgtk\" (UniqueName: \"kubernetes.io/projected/6ff44b73-9f64-4fca-95f8-6a5a24540843-kube-api-access-kwgtk\") pod \"network-metrics-daemon-plsqk\" (UID: \"6ff44b73-9f64-4fca-95f8-6a5a24540843\") " pod="openshift-multus/network-metrics-daemon-plsqk" Oct 30 01:29:31.765513 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765449 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-kubelet\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.765513 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765477 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-run-openvswitch\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.765513 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765494 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-socket-dir-parent\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.765513 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765509 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-run-k8s-cni-cncf-io\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765524 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-run-ovn\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765541 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-system-cni-dir\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765558 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-run-openvswitch\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765563 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-etc-kubernetes\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765572 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-socket-dir-parent\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765580 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-run-k8s-cni-cncf-io\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765595 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/91d108c9-4e8c-4d00-8e42-1eecd52741a2-crio-socket\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765600 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-run-ovn\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765616 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-kubelet\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765623 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-run-netns\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765600 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-etc-kubernetes\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765651 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-run-netns\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765658 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-system-cni-dir\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765675 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40d2d25c-e7a0-4ded-9674-19c6d371522e-system-cni-dir\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765698 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4-host-slash\") pod \"iptables-alerter-p9b7v\" (UID: \"07c8420b-05bf-4d3e-a1a2-7a6d55fffad4\") " pod="openshift-network-operator/iptables-alerter-p9b7v" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765713 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-run-systemd\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765711 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-run-netns\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765712 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/40d2d25c-e7a0-4ded-9674-19c6d371522e-system-cni-dir\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765731 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-sysconfig\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765662 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/91d108c9-4e8c-4d00-8e42-1eecd52741a2-crio-socket\") pod \"insights-runtime-extractor-vccw5\" (UID: \"91d108c9-4e8c-4d00-8e42-1eecd52741a2\") " pod="openshift-insights/insights-runtime-extractor-vccw5" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765757 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-lib-modules\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.765766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765776 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-run-netns\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765781 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07c8420b-05bf-4d3e-a1a2-7a6d55fffad4-host-slash\") pod \"iptables-alerter-p9b7v\" (UID: \"07c8420b-05bf-4d3e-a1a2-7a6d55fffad4\") " pod="openshift-network-operator/iptables-alerter-p9b7v" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765790 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-log-socket\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765802 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-sysconfig\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765791 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-run-systemd\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765815 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-log-socket\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765829 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-run\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765849 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-slash\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765867 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-cni-dir\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765885 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-host\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765896 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-run\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765885 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-lib-modules\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765912 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f486e9e-aa32-4a9a-877b-eca8dbba7091-host\") pod \"node-ca-lq7lj\" (UID: \"4f486e9e-aa32-4a9a-877b-eca8dbba7091\") " pod="openshift-image-registry/node-ca-lq7lj" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765933 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40d2d25c-e7a0-4ded-9674-19c6d371522e-cnibin\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765935 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-host\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765914 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-slash\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765948 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-var-lib-kubelet\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765950 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-cni-dir\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765965 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40d2d25c-e7a0-4ded-9674-19c6d371522e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765970 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/40d2d25c-e7a0-4ded-9674-19c6d371522e-cnibin\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765937 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f486e9e-aa32-4a9a-877b-eca8dbba7091-host\") pod \"node-ca-lq7lj\" (UID: \"4f486e9e-aa32-4a9a-877b-eca8dbba7091\") " pod="openshift-image-registry/node-ca-lq7lj" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.765987 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-modprobe-d\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766011 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-var-lib-kubelet\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766032 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/40d2d25c-e7a0-4ded-9674-19c6d371522e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zrv6d\" (UID: \"40d2d25c-e7a0-4ded-9674-19c6d371522e\") " pod="openshift-multus/multus-additional-cni-plugins-zrv6d" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766047 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-etc-openvswitch\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766014 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-etc-openvswitch\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766082 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-socket-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766125 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-run-ovn-kubernetes\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766134 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-modprobe-d\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766144 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4c33058-34c5-49f3-b803-984c54ff5dad-sys\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766150 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-socket-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766162 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766181 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-run-ovn-kubernetes\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766188 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-dbus\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766194 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4c33058-34c5-49f3-b803-984c54ff5dad-sys\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766219 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-kubernetes\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766239 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-dbus\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766243 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766247 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-var-lib-kubelet\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766272 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-hostroot\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766296 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-cni-bin\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766300 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-var-lib-kubelet\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766288 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-kubernetes\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766322 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-var-lib-cni-bin\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766326 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-hostroot\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766342 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-cni-bin\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766352 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-sysctl-d\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766357 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-var-lib-cni-bin\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766380 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/174be2db-9c13-45fe-b91f-c3302ecb57e2-hosts-file\") pod \"node-resolver-8wd8v\" (UID: \"174be2db-9c13-45fe-b91f-c3302ecb57e2\") " pod="openshift-dns/node-resolver-8wd8v" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766394 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-conf-dir\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766410 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-cni-netd\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766434 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-multus-conf-dir\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766446 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/174be2db-9c13-45fe-b91f-c3302ecb57e2-hosts-file\") pod \"node-resolver-8wd8v\" (UID: \"174be2db-9c13-45fe-b91f-c3302ecb57e2\") " pod="openshift-dns/node-resolver-8wd8v" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766454 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-sysctl-d\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766432 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-var-lib-cni-multus\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766477 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-cni-netd\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766484 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-sys\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766483 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-var-lib-cni-multus\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766519 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-sys\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766517 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-systemd\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766544 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-registration-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766552 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-systemd\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766563 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766584 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-registration-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766587 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-wtmp\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766610 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-var-lib-openvswitch\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766614 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b4c33058-34c5-49f3-b803-984c54ff5dad-node-exporter-wtmp\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766612 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766632 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-kubelet-config\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766638 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-var-lib-openvswitch\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766651 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-systemd-units\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766665 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-node-log\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.766634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766685 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e0bbf87d-7baa-4b99-9b80-695c2c99c4c8-kubelet-config\") pod \"global-pull-secret-syncer-55kfb\" (UID: \"e0bbf87d-7baa-4b99-9b80-695c2c99c4c8\") " pod="kube-system/global-pull-secret-syncer-55kfb" Oct 30 01:29:31.769126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766697 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-device-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.769126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766713 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-node-log\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.769126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766727 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c01f6b4a-5666-4734-b403-3818692c3a74-systemd-units\") pod \"ovnkube-node-4qqkx\" (UID: \"c01f6b4a-5666-4734-b403-3818692c3a74\") " pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:31.769126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766730 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-sys-fs\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.769126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766755 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-device-dir\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.769126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766767 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b4c33058-34c5-49f3-b803-984c54ff5dad-root\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.769126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766770 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-sys-fs\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.769126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766791 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.769126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766809 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-run-multus-certs\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.769126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766793 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b4c33058-34c5-49f3-b803-984c54ff5dad-root\") pod \"node-exporter-nd754\" (UID: \"b4c33058-34c5-49f3-b803-984c54ff5dad\") " pod="openshift-monitoring/node-exporter-nd754" Oct 30 01:29:31.769126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766856 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d50883a3-5056-4826-884e-90fe75441e5d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hls4s\" (UID: \"d50883a3-5056-4826-884e-90fe75441e5d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hls4s" Oct 30 01:29:31.769126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766824 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-cnibin\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.769126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766861 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-host-run-multus-certs\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.769126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766855 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/18cee84e-69dd-4296-98d7-de2e6e0c6fe0-cnibin\") pod \"multus-sgm2w\" (UID: \"18cee84e-69dd-4296-98d7-de2e6e0c6fe0\") " pod="openshift-multus/multus-sgm2w" Oct 30 01:29:31.769126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.766892 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-sysctl-conf\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:31.769126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:31.767039 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d1facbe7-deb0-4fec-94de-42a0dd91ff4f-etc-sysctl-conf\") pod \"tuned-kjlnq\" (UID: \"d1facbe7-deb0-4fec-94de-42a0dd91ff4f\") " pod="openshift-cluster-node-tuning-operator/tuned-kjlnq" Oct 30 01:29:32.011169 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:32.011141 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:32.881684 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:32.881643 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:32.990815 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:32.990785 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6vkc5" Oct 30 01:29:32.996813 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:32.996782 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:33.159852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:33.159774 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:33.599265 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:33.599238 5998 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:29:33.748246 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:33.748218 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:33.806202 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:33.806166 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:29:33.982548 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:33.982518 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-868b7dc86d-hzrmz" Oct 30 01:29:34.349791 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:34.349709 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4qqkx" Oct 30 01:29:36.146109 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:36.146060 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-767946cb5-q64ns" Oct 30 01:29:38.028456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:38.028422 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:38.141322 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:38.141286 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:38.866719 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:38.866685 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:39.097854 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:39.097818 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:41.219541 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:41.219508 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:41.325390 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:41.325337 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:41.433517 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:41.433486 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:43.051937 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:43.051868 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:43.156812 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:43.156778 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:44.867997 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:44.867963 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 01:29:44.868454 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:44.868325 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="prometheus" containerID="cri-o://8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754" gracePeriod=600 Oct 30 01:29:44.868454 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:44.868354 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="thanos-sidecar" containerID="cri-o://9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe" gracePeriod=600 Oct 30 01:29:44.868454 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:44.868374 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e" gracePeriod=600 Oct 30 01:29:44.868454 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:44.868387 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="config-reloader" containerID="cri-o://020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac" gracePeriod=600 Oct 30 01:29:44.868454 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:44.868402 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="kube-rbac-proxy-web" containerID="cri-o://e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e" gracePeriod=600 Oct 30 01:29:44.868454 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:44.868334 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="kube-rbac-proxy" containerID="cri-o://bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe" gracePeriod=600 Oct 30 01:29:45.129590 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.129519 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.269982 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.269948 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-kube-api-access-wspxg\" (UniqueName: \"kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-kube-api-access-wspxg\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270227 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270001 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-web-config\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-web-config\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270227 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270032 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-kubelet-serving-ca-bundle\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270227 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270061 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-0\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270227 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270099 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-1\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270227 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270126 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-serving-certs-ca-bundle\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270227 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270160 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-trusted-ca-bundle\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270227 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270186 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-metrics-client-certs\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270227 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270220 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270703 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270252 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-tls-assets\" (UniqueName: \"kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-tls-assets\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270703 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270320 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-tls\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270703 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270378 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-db\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270703 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270413 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270703 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270464 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-metrics-client-ca\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270703 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270499 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-grpc-tls\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270703 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270494 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-kubelet-serving-ca-bundle" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:45.270703 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270534 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-2\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270703 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270583 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-thanos-prometheus-http-client-file\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270703 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270613 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270703 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270659 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config-out\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.270703 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270688 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-kube-rbac-proxy\") pod \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\" (UID: \"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047\") " Oct 30 01:29:45.271321 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270781 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-1" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:45.271321 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270936 5998 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.271321 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.270959 5998 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.271478 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.271384 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-trusted-ca-bundle" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:45.271799 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.271762 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-0" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:45.271999 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.271975 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-metrics-client-ca" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:45.272439 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.272413 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-serving-certs-ca-bundle" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:45.272593 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.272541 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-2" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:29:45.273363 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.273279 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-db" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:45.273733 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.273697 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-kube-api-access-wspxg" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "kube-api-access-wspxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:45.273883 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.273696 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-thanos-sidecar-tls" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:45.273990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.273922 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-kube-rbac-proxy" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:45.274539 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.274488 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-grpc-tls" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:45.274851 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.274822 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-thanos-prometheus-http-client-file" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:45.275112 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.275060 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-tls-assets" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:29:45.275112 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.275070 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:45.275295 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.275251 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-metrics-client-certs" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:45.275352 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.275321 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config-out" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:29:45.276278 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.276244 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-tls" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:45.276771 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.276750 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-kube-rbac-proxy-web" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:45.283687 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.283665 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-web-config" pod "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" (UID: "8e3a6b92-65a9-411e-bbf0-bf12b5c9c047"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:29:45.371652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.371616 5998 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.371652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.371648 5998 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-metrics-client-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.371919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.371662 5998 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-grpc-tls\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.371919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.371675 5998 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.371919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.371687 5998 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.371919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.371699 5998 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.371919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.371710 5998 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-config-out\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.371919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.371723 5998 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-kube-rbac-proxy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.371919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.371734 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wspxg\" (UniqueName: \"kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-kube-api-access-wspxg\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.371919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.371746 5998 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-web-config\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.371919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.371757 5998 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.371919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.371770 5998 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.371919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.371782 5998 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.371919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.371795 5998 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-metrics-client-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.371919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.371808 5998 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.371919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.371822 5998 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-tls-assets\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.371919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.371835 5998 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.371919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.371847 5998 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047-prometheus-k8s-db\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:29:45.642256 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.642222 5998 generic.go:356] "Generic (PLEG): container finished" podID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerID="b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e" exitCode=0 Oct 30 01:29:45.642256 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.642249 5998 generic.go:356] "Generic (PLEG): container finished" podID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerID="bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe" exitCode=0 Oct 30 01:29:45.642256 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.642257 5998 generic.go:356] "Generic (PLEG): container finished" podID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerID="e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e" exitCode=0 Oct 30 01:29:45.642256 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.642263 5998 generic.go:356] "Generic (PLEG): container finished" podID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerID="9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe" exitCode=0 Oct 30 01:29:45.642609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.642269 5998 generic.go:356] "Generic (PLEG): container finished" podID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerID="020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac" exitCode=0 Oct 30 01:29:45.642609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.642274 5998 generic.go:356] "Generic (PLEG): container finished" podID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerID="8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754" exitCode=0 Oct 30 01:29:45.642609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.642302 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.642609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.642310 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047","Type":"ContainerDied","Data":"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e"} Oct 30 01:29:45.642609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.642391 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047","Type":"ContainerDied","Data":"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe"} Oct 30 01:29:45.642609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.642420 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047","Type":"ContainerDied","Data":"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e"} Oct 30 01:29:45.642609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.642447 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047","Type":"ContainerDied","Data":"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe"} Oct 30 01:29:45.642609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.642456 5998 scope.go:122] "RemoveContainer" containerID="b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e" Oct 30 01:29:45.642609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.642478 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047","Type":"ContainerDied","Data":"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac"} Oct 30 01:29:45.642609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.642508 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047","Type":"ContainerDied","Data":"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754"} Oct 30 01:29:45.642609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.642535 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e3a6b92-65a9-411e-bbf0-bf12b5c9c047","Type":"ContainerDied","Data":"de46b4224869ff221f36081f4ff9119a601df3ef83012041978fcab90cafee19"} Oct 30 01:29:45.650522 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.650507 5998 scope.go:122] "RemoveContainer" containerID="bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe" Oct 30 01:29:45.657407 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.657388 5998 scope.go:122] "RemoveContainer" containerID="e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e" Oct 30 01:29:45.663871 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.663851 5998 scope.go:122] "RemoveContainer" containerID="9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe" Oct 30 01:29:45.670266 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.670243 5998 scope.go:122] "RemoveContainer" containerID="020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac" Oct 30 01:29:45.677205 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.677183 5998 scope.go:122] "RemoveContainer" containerID="8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754" Oct 30 01:29:45.684293 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.684270 5998 scope.go:122] "RemoveContainer" containerID="a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e" Oct 30 01:29:45.690838 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.690820 5998 scope.go:122] "RemoveContainer" containerID="b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e" Oct 30 01:29:45.691134 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:29:45.691111 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e\": container with ID starting with b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e not found: ID does not exist" containerID="b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e" Oct 30 01:29:45.691220 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.691148 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e"} err="failed to get container status \"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e\": rpc error: code = NotFound desc = could not find container \"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e\": container with ID starting with b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e not found: ID does not exist" Oct 30 01:29:45.691220 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.691192 5998 scope.go:122] "RemoveContainer" containerID="bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe" Oct 30 01:29:45.691436 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:29:45.691415 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe\": container with ID starting with bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe not found: ID does not exist" containerID="bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe" Oct 30 01:29:45.691493 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.691443 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe"} err="failed to get container status \"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe\": rpc error: code = NotFound desc = could not find container \"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe\": container with ID starting with bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe not found: ID does not exist" Oct 30 01:29:45.691493 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.691458 5998 scope.go:122] "RemoveContainer" containerID="e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e" Oct 30 01:29:45.691669 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:29:45.691644 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e\": container with ID starting with e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e not found: ID does not exist" containerID="e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e" Oct 30 01:29:45.691715 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.691668 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e"} err="failed to get container status \"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e\": rpc error: code = NotFound desc = could not find container \"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e\": container with ID starting with e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e not found: ID does not exist" Oct 30 01:29:45.691715 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.691686 5998 scope.go:122] "RemoveContainer" containerID="9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe" Oct 30 01:29:45.691872 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:29:45.691851 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe\": container with ID starting with 9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe not found: ID does not exist" containerID="9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe" Oct 30 01:29:45.691910 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.691877 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe"} err="failed to get container status \"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe\": rpc error: code = NotFound desc = could not find container \"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe\": container with ID starting with 9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe not found: ID does not exist" Oct 30 01:29:45.691910 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.691889 5998 scope.go:122] "RemoveContainer" containerID="020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac" Oct 30 01:29:45.692068 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:29:45.692050 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac\": container with ID starting with 020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac not found: ID does not exist" containerID="020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac" Oct 30 01:29:45.692127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.692074 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac"} err="failed to get container status \"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac\": rpc error: code = NotFound desc = could not find container \"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac\": container with ID starting with 020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac not found: ID does not exist" Oct 30 01:29:45.692127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.692108 5998 scope.go:122] "RemoveContainer" containerID="8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754" Oct 30 01:29:45.692326 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:29:45.692308 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754\": container with ID starting with 8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754 not found: ID does not exist" containerID="8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754" Oct 30 01:29:45.692388 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.692334 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754"} err="failed to get container status \"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754\": rpc error: code = NotFound desc = could not find container \"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754\": container with ID starting with 8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754 not found: ID does not exist" Oct 30 01:29:45.692388 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.692355 5998 scope.go:122] "RemoveContainer" containerID="a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e" Oct 30 01:29:45.692578 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:29:45.692560 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e\": container with ID starting with a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e not found: ID does not exist" containerID="a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e" Oct 30 01:29:45.692618 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.692583 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e"} err="failed to get container status \"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e\": rpc error: code = NotFound desc = could not find container \"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e\": container with ID starting with a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e not found: ID does not exist" Oct 30 01:29:45.692618 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.692597 5998 scope.go:122] "RemoveContainer" containerID="b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e" Oct 30 01:29:45.692813 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.692797 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e"} err="failed to get container status \"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e\": rpc error: code = NotFound desc = could not find container \"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e\": container with ID starting with b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e not found: ID does not exist" Oct 30 01:29:45.692864 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.692816 5998 scope.go:122] "RemoveContainer" containerID="bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe" Oct 30 01:29:45.692998 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.692981 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe"} err="failed to get container status \"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe\": rpc error: code = NotFound desc = could not find container \"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe\": container with ID starting with bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe not found: ID does not exist" Oct 30 01:29:45.693048 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.693000 5998 scope.go:122] "RemoveContainer" containerID="e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e" Oct 30 01:29:45.693217 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.693199 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e"} err="failed to get container status \"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e\": rpc error: code = NotFound desc = could not find container \"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e\": container with ID starting with e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e not found: ID does not exist" Oct 30 01:29:45.693266 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.693218 5998 scope.go:122] "RemoveContainer" containerID="9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe" Oct 30 01:29:45.693412 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.693394 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe"} err="failed to get container status \"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe\": rpc error: code = NotFound desc = could not find container \"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe\": container with ID starting with 9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe not found: ID does not exist" Oct 30 01:29:45.693477 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.693413 5998 scope.go:122] "RemoveContainer" containerID="020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac" Oct 30 01:29:45.693613 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.693597 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac"} err="failed to get container status \"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac\": rpc error: code = NotFound desc = could not find container \"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac\": container with ID starting with 020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac not found: ID does not exist" Oct 30 01:29:45.693684 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.693614 5998 scope.go:122] "RemoveContainer" containerID="8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754" Oct 30 01:29:45.693862 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.693845 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754"} err="failed to get container status \"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754\": rpc error: code = NotFound desc = could not find container \"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754\": container with ID starting with 8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754 not found: ID does not exist" Oct 30 01:29:45.693925 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.693862 5998 scope.go:122] "RemoveContainer" containerID="a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e" Oct 30 01:29:45.694106 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.694071 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e"} err="failed to get container status \"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e\": rpc error: code = NotFound desc = could not find container \"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e\": container with ID starting with a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e not found: ID does not exist" Oct 30 01:29:45.694161 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.694108 5998 scope.go:122] "RemoveContainer" containerID="b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e" Oct 30 01:29:45.694329 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.694311 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e"} err="failed to get container status \"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e\": rpc error: code = NotFound desc = could not find container \"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e\": container with ID starting with b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e not found: ID does not exist" Oct 30 01:29:45.694405 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.694330 5998 scope.go:122] "RemoveContainer" containerID="bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe" Oct 30 01:29:45.694589 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.694567 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe"} err="failed to get container status \"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe\": rpc error: code = NotFound desc = could not find container \"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe\": container with ID starting with bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe not found: ID does not exist" Oct 30 01:29:45.694629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.694590 5998 scope.go:122] "RemoveContainer" containerID="e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e" Oct 30 01:29:45.694828 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.694810 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e"} err="failed to get container status \"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e\": rpc error: code = NotFound desc = could not find container \"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e\": container with ID starting with e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e not found: ID does not exist" Oct 30 01:29:45.694873 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.694828 5998 scope.go:122] "RemoveContainer" containerID="9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe" Oct 30 01:29:45.695054 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.695037 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe"} err="failed to get container status \"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe\": rpc error: code = NotFound desc = could not find container \"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe\": container with ID starting with 9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe not found: ID does not exist" Oct 30 01:29:45.695115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.695054 5998 scope.go:122] "RemoveContainer" containerID="020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac" Oct 30 01:29:45.695271 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.695251 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac"} err="failed to get container status \"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac\": rpc error: code = NotFound desc = could not find container \"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac\": container with ID starting with 020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac not found: ID does not exist" Oct 30 01:29:45.695271 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.695268 5998 scope.go:122] "RemoveContainer" containerID="8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754" Oct 30 01:29:45.695498 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.695479 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754"} err="failed to get container status \"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754\": rpc error: code = NotFound desc = could not find container \"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754\": container with ID starting with 8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754 not found: ID does not exist" Oct 30 01:29:45.695550 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.695500 5998 scope.go:122] "RemoveContainer" containerID="a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e" Oct 30 01:29:45.695696 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.695677 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e"} err="failed to get container status \"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e\": rpc error: code = NotFound desc = could not find container \"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e\": container with ID starting with a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e not found: ID does not exist" Oct 30 01:29:45.695742 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.695697 5998 scope.go:122] "RemoveContainer" containerID="b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e" Oct 30 01:29:45.695902 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.695883 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e"} err="failed to get container status \"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e\": rpc error: code = NotFound desc = could not find container \"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e\": container with ID starting with b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e not found: ID does not exist" Oct 30 01:29:45.695946 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.695902 5998 scope.go:122] "RemoveContainer" containerID="bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe" Oct 30 01:29:45.696113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.696076 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe"} err="failed to get container status \"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe\": rpc error: code = NotFound desc = could not find container \"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe\": container with ID starting with bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe not found: ID does not exist" Oct 30 01:29:45.696172 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.696112 5998 scope.go:122] "RemoveContainer" containerID="e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e" Oct 30 01:29:45.696310 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.696291 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e"} err="failed to get container status \"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e\": rpc error: code = NotFound desc = could not find container \"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e\": container with ID starting with e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e not found: ID does not exist" Oct 30 01:29:45.696375 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.696313 5998 scope.go:122] "RemoveContainer" containerID="9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe" Oct 30 01:29:45.696568 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.696529 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe"} err="failed to get container status \"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe\": rpc error: code = NotFound desc = could not find container \"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe\": container with ID starting with 9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe not found: ID does not exist" Oct 30 01:29:45.696568 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.696546 5998 scope.go:122] "RemoveContainer" containerID="020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac" Oct 30 01:29:45.696748 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.696734 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac"} err="failed to get container status \"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac\": rpc error: code = NotFound desc = could not find container \"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac\": container with ID starting with 020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac not found: ID does not exist" Oct 30 01:29:45.696792 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.696749 5998 scope.go:122] "RemoveContainer" containerID="8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754" Oct 30 01:29:45.696963 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.696946 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754"} err="failed to get container status \"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754\": rpc error: code = NotFound desc = could not find container \"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754\": container with ID starting with 8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754 not found: ID does not exist" Oct 30 01:29:45.697004 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.696964 5998 scope.go:122] "RemoveContainer" containerID="a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e" Oct 30 01:29:45.697220 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.697204 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e"} err="failed to get container status \"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e\": rpc error: code = NotFound desc = could not find container \"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e\": container with ID starting with a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e not found: ID does not exist" Oct 30 01:29:45.697274 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.697221 5998 scope.go:122] "RemoveContainer" containerID="b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e" Oct 30 01:29:45.697427 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.697413 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e"} err="failed to get container status \"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e\": rpc error: code = NotFound desc = could not find container \"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e\": container with ID starting with b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e not found: ID does not exist" Oct 30 01:29:45.697469 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.697428 5998 scope.go:122] "RemoveContainer" containerID="bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe" Oct 30 01:29:45.697626 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.697611 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe"} err="failed to get container status \"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe\": rpc error: code = NotFound desc = could not find container \"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe\": container with ID starting with bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe not found: ID does not exist" Oct 30 01:29:45.697662 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.697626 5998 scope.go:122] "RemoveContainer" containerID="e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e" Oct 30 01:29:45.697822 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.697807 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e"} err="failed to get container status \"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e\": rpc error: code = NotFound desc = could not find container \"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e\": container with ID starting with e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e not found: ID does not exist" Oct 30 01:29:45.697861 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.697822 5998 scope.go:122] "RemoveContainer" containerID="9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe" Oct 30 01:29:45.698014 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.697999 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe"} err="failed to get container status \"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe\": rpc error: code = NotFound desc = could not find container \"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe\": container with ID starting with 9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe not found: ID does not exist" Oct 30 01:29:45.698014 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.698013 5998 scope.go:122] "RemoveContainer" containerID="020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac" Oct 30 01:29:45.698245 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.698226 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac"} err="failed to get container status \"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac\": rpc error: code = NotFound desc = could not find container \"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac\": container with ID starting with 020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac not found: ID does not exist" Oct 30 01:29:45.698316 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.698246 5998 scope.go:122] "RemoveContainer" containerID="8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754" Oct 30 01:29:45.698472 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.698456 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754"} err="failed to get container status \"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754\": rpc error: code = NotFound desc = could not find container \"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754\": container with ID starting with 8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754 not found: ID does not exist" Oct 30 01:29:45.698472 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.698472 5998 scope.go:122] "RemoveContainer" containerID="a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e" Oct 30 01:29:45.698670 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.698648 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e"} err="failed to get container status \"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e\": rpc error: code = NotFound desc = could not find container \"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e\": container with ID starting with a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e not found: ID does not exist" Oct 30 01:29:45.698670 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.698664 5998 scope.go:122] "RemoveContainer" containerID="b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e" Oct 30 01:29:45.698849 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.698835 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e"} err="failed to get container status \"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e\": rpc error: code = NotFound desc = could not find container \"b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e\": container with ID starting with b225ad857931d396eecba4870d4f7e879c6c9d1f488a0f0841f1a21fe052da1e not found: ID does not exist" Oct 30 01:29:45.698891 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.698849 5998 scope.go:122] "RemoveContainer" containerID="bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe" Oct 30 01:29:45.699011 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.698996 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe"} err="failed to get container status \"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe\": rpc error: code = NotFound desc = could not find container \"bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe\": container with ID starting with bb2ccbf9147ce24923e984b148d832430603143675a5a17a32aad1f6c84769fe not found: ID does not exist" Oct 30 01:29:45.699052 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.699011 5998 scope.go:122] "RemoveContainer" containerID="e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e" Oct 30 01:29:45.699247 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.699230 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e"} err="failed to get container status \"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e\": rpc error: code = NotFound desc = could not find container \"e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e\": container with ID starting with e4eb565ea7eaaf2573ae863de829b9e968fa2c15bbba15d5924ca08a3a03803e not found: ID does not exist" Oct 30 01:29:45.699294 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.699247 5998 scope.go:122] "RemoveContainer" containerID="9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe" Oct 30 01:29:45.699459 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.699444 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe"} err="failed to get container status \"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe\": rpc error: code = NotFound desc = could not find container \"9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe\": container with ID starting with 9d6bec69e020c5b0ba93b35cc3b05659f69f1ce176b8d6340926b52dcb47dbfe not found: ID does not exist" Oct 30 01:29:45.699500 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.699459 5998 scope.go:122] "RemoveContainer" containerID="020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac" Oct 30 01:29:45.699649 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.699634 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac"} err="failed to get container status \"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac\": rpc error: code = NotFound desc = could not find container \"020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac\": container with ID starting with 020e0ab2be14d39e4e85cf88d00838ea9c3275ebe25398e21a4d3b0b932ae7ac not found: ID does not exist" Oct 30 01:29:45.699691 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.699649 5998 scope.go:122] "RemoveContainer" containerID="8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754" Oct 30 01:29:45.699815 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.699800 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754"} err="failed to get container status \"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754\": rpc error: code = NotFound desc = could not find container \"8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754\": container with ID starting with 8d48b75992b46557dddd342ab1a6fe33c87b064178b40159886f3c77af9ac754 not found: ID does not exist" Oct 30 01:29:45.699856 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.699814 5998 scope.go:122] "RemoveContainer" containerID="a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e" Oct 30 01:29:45.700044 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.700018 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e"} err="failed to get container status \"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e\": rpc error: code = NotFound desc = could not find container \"a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e\": container with ID starting with a00386496520839686dcd14c55a7b4eb7d3752f1ebb831dbee8f885f1b5de52e not found: ID does not exist" Oct 30 01:29:45.707005 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.706985 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 01:29:45.718689 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.718668 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 01:29:45.744595 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744572 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 01:29:45.744809 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744798 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="kube-rbac-proxy-web" Oct 30 01:29:45.744852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744810 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="kube-rbac-proxy-web" Oct 30 01:29:45.744852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744821 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="thanos-sidecar" Oct 30 01:29:45.744852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744826 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="thanos-sidecar" Oct 30 01:29:45.744852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744833 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="prometheus" Oct 30 01:29:45.744852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744837 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="prometheus" Oct 30 01:29:45.744852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744847 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="kube-rbac-proxy" Oct 30 01:29:45.744852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744852 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="kube-rbac-proxy" Oct 30 01:29:45.745051 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744864 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="kube-rbac-proxy-thanos" Oct 30 01:29:45.745051 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744869 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="kube-rbac-proxy-thanos" Oct 30 01:29:45.745051 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744876 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="config-reloader" Oct 30 01:29:45.745051 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744880 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="config-reloader" Oct 30 01:29:45.745051 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744889 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="init-config-reloader" Oct 30 01:29:45.745051 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744894 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="init-config-reloader" Oct 30 01:29:45.745051 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744930 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="kube-rbac-proxy" Oct 30 01:29:45.745051 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744938 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="kube-rbac-proxy-thanos" Oct 30 01:29:45.745051 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744945 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="config-reloader" Oct 30 01:29:45.745051 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744952 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="init-config-reloader" Oct 30 01:29:45.745051 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744959 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="kube-rbac-proxy-web" Oct 30 01:29:45.745051 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744964 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="thanos-sidecar" Oct 30 01:29:45.745051 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.744969 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" containerName="prometheus" Oct 30 01:29:45.778553 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.778532 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.783867 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.783845 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 01:29:45.794572 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.794547 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 01:29:45.795083 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.795065 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 01:29:45.795083 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.795080 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 01:29:45.795245 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.795070 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 01:29:45.795245 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.795211 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 01:29:45.795344 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.795282 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 01:29:45.795344 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.795314 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 01:29:45.795344 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.795317 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 01:29:45.795344 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.795335 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 01:29:45.795490 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.795289 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 01:29:45.795548 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.795534 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 01:29:45.795668 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.795654 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nw28w\"" Oct 30 01:29:45.808076 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.808052 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 01:29:45.808076 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.808065 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-569c19rarmjmk\"" Oct 30 01:29:45.808212 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.808077 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 01:29:45.808267 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.808249 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 01:29:45.898132 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.898105 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-44ftt" Oct 30 01:29:45.976670 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.976637 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0b5239b-d668-4a7a-b836-9bebc978f33c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.976837 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.976681 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.976837 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.976703 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.976837 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.976783 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.977012 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.976849 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.977012 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.976879 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.977012 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.976897 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnmfv\" (UniqueName: \"kubernetes.io/projected/c0b5239b-d668-4a7a-b836-9bebc978f33c-kube-api-access-pnmfv\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.977012 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.976940 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0b5239b-d668-4a7a-b836-9bebc978f33c-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.977012 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.976992 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.977216 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.977050 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.977216 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.977084 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.977216 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.977151 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.977216 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.977169 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.977344 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.977218 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.977344 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.977234 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.977344 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.977251 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.977344 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.977268 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-config\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.977344 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.977303 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.977508 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.977354 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:45.977508 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:45.977376 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0b5239b-d668-4a7a-b836-9bebc978f33c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078149 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078105 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078332 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078163 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078332 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078188 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078332 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078215 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078332 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078235 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnmfv\" (UniqueName: \"kubernetes.io/projected/c0b5239b-d668-4a7a-b836-9bebc978f33c-kube-api-access-pnmfv\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078332 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078261 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0b5239b-d668-4a7a-b836-9bebc978f33c-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078332 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078286 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078332 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078310 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078334 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078384 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078412 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078463 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078491 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078522 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078556 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-config\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078563 5998 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 01:29:46.078693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078581 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078609 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078657 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0b5239b-d668-4a7a-b836-9bebc978f33c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.078693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078682 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0b5239b-d668-4a7a-b836-9bebc978f33c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.079290 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078733 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.079290 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.078996 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.083138 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.079948 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.083138 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.080078 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.083138 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.080438 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.083138 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.081561 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.083138 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.081764 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.083138 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.082959 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.083429 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.083173 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-web-config\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.087222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.083524 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.087222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.083578 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.087222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.083720 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0b5239b-d668-4a7a-b836-9bebc978f33c-config-out\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.087222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.083817 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.087222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.084200 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.087222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.084480 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c0b5239b-d668-4a7a-b836-9bebc978f33c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.087222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.084651 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c0b5239b-d668-4a7a-b836-9bebc978f33c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.087222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.085633 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.087222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.086828 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0b5239b-d668-4a7a-b836-9bebc978f33c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.087222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.087177 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.089337 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.089320 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c0b5239b-d668-4a7a-b836-9bebc978f33c-config\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.101490 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.101466 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnmfv\" (UniqueName: \"kubernetes.io/projected/c0b5239b-d668-4a7a-b836-9bebc978f33c-kube-api-access-pnmfv\") pod \"prometheus-k8s-0\" (UID: \"c0b5239b-d668-4a7a-b836-9bebc978f33c\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.185532 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.185498 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:46.536290 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.536199 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e3a6b92-65a9-411e-bbf0-bf12b5c9c047" path="/var/lib/kubelet/pods/8e3a6b92-65a9-411e-bbf0-bf12b5c9c047/volumes" Oct 30 01:29:46.550171 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.550146 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 01:29:46.556135 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:29:46.556102 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b5239b_d668_4a7a_b836_9bebc978f33c.slice/crio-7ae2340b013277b64b7e9ad16740872da2be91cb2b26c7f984e25516f2c8ca15 WatchSource:0}: Error finding container 7ae2340b013277b64b7e9ad16740872da2be91cb2b26c7f984e25516f2c8ca15: Status 404 returned error can't find the container with id 7ae2340b013277b64b7e9ad16740872da2be91cb2b26c7f984e25516f2c8ca15 Oct 30 01:29:46.647701 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.647669 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0b5239b-d668-4a7a-b836-9bebc978f33c","Type":"ContainerStarted","Data":"b636135a453c77840c2dce832073db8e23520e0332db83f4d633f7c3eacd4c91"} Oct 30 01:29:46.647836 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:46.647749 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0b5239b-d668-4a7a-b836-9bebc978f33c","Type":"ContainerStarted","Data":"7ae2340b013277b64b7e9ad16740872da2be91cb2b26c7f984e25516f2c8ca15"} Oct 30 01:29:47.653264 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:47.653223 5998 generic.go:356] "Generic (PLEG): container finished" podID="c0b5239b-d668-4a7a-b836-9bebc978f33c" containerID="b636135a453c77840c2dce832073db8e23520e0332db83f4d633f7c3eacd4c91" exitCode=0 Oct 30 01:29:47.653739 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:47.653310 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0b5239b-d668-4a7a-b836-9bebc978f33c","Type":"ContainerDied","Data":"b636135a453c77840c2dce832073db8e23520e0332db83f4d633f7c3eacd4c91"} Oct 30 01:29:48.660145 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:48.660042 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0b5239b-d668-4a7a-b836-9bebc978f33c","Type":"ContainerStarted","Data":"34e72772cfb2f60fce989a0a22f2800aceba757e5f57a541c1add4b11172db6b"} Oct 30 01:29:48.660145 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:48.660132 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0b5239b-d668-4a7a-b836-9bebc978f33c","Type":"ContainerStarted","Data":"955948c9654510788bb78b1dc324e74742159acf383f4e5f8e06ac8927d999f2"} Oct 30 01:29:48.660519 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:48.660166 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0b5239b-d668-4a7a-b836-9bebc978f33c","Type":"ContainerStarted","Data":"4a3f5db26e102ecb2f9b252bd5ce5c34a2fa54de5d34f8375330155adad46296"} Oct 30 01:29:48.660519 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:48.660197 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0b5239b-d668-4a7a-b836-9bebc978f33c","Type":"ContainerStarted","Data":"2202ecb2c825bc366da856d19e386b6838af4df739cac7ec0678eb994d9dee5d"} Oct 30 01:29:48.660519 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:48.660229 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0b5239b-d668-4a7a-b836-9bebc978f33c","Type":"ContainerStarted","Data":"9d61199ef65596830aa91bec110c07aecd17a30f0b1b1805301b3ddfc0959cd5"} Oct 30 01:29:48.660519 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:48.660267 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c0b5239b-d668-4a7a-b836-9bebc978f33c","Type":"ContainerStarted","Data":"f6a9f91ee4e3500c23d8097d247ffa783fdb88d8ac7d9d3f31eea0529a6e80c3"} Oct 30 01:29:48.717505 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:48.717451 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.688917053 podStartE2EDuration="3.717438252s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.521199ms" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:48.716567903 +0000 UTC m=+18.295359018" watchObservedRunningTime="2026-10-30 01:29:48.717438252 +0000 UTC m=+18.296229327" Oct 30 01:29:50.031885 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:50.031853 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:50.137775 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:50.137739 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:51.186218 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:51.186183 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:52.034106 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:52.034061 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:52.156959 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:52.156925 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:54.039497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:54.039454 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:54.165777 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:54.165745 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:29:55.214132 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:29:55.214080 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:29:55.947528 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:55.947483 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:55.963071 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:55.963045 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:29:56.200976 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:29:56.200900 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 01:30:00.599343 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:00.599307 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:03.243010 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:03.242975 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:03.355077 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:03.355046 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:03.809073 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:03.809033 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pct74" Oct 30 01:30:04.810817 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:04.810782 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:04.916913 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:04.916878 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:11.570803 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:11.570766 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:11.678324 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:11.678287 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:11.877550 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:11.877461 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:11.984787 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:11.984753 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:17.803620 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:17.803590 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:17.929805 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:17.929767 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:20.494988 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:20.494947 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:20.600528 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:20.600493 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:25.215581 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:30:25.215536 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:30:27.719180 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:27.719143 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:27.825791 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:27.825760 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:31.376288 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:31.376248 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:32.807138 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:32.807102 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:32.909690 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:32.909625 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:46.459947 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:46.459912 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:46.567318 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:30:46.567279 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:30:55.215656 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:30:55.215620 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:31:25.215527 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:31:25.215486 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:31:30.602312 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:31:30.602270 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:31.499033 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:31:31.498998 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:32.862740 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:31:32.862699 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:35.992644 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:31:35.992610 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:36.096797 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:31:36.096761 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:31:55.215126 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:31:55.215073 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:32:25.215516 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:32:25.215477 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:32:29.882029 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:29.881991 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:31.631825 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:31.631786 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:45.893812 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:45.893736 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:46.055557 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:46.055519 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:47.546214 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:47.546183 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:47.713645 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:47.713616 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:48.823750 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.823716 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r"] Oct 30 01:32:48.826662 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.826637 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r" Oct 30 01:32:48.842045 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.842025 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-5r7vv\"" Oct 30 01:32:48.842510 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.842496 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 01:32:48.846775 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.846759 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v62h2\" (UniqueName: \"kubernetes.io/projected/ba7596b7-01a9-412d-a39d-ef94ea4c7103-kube-api-access-v62h2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r\" (UID: \"ba7596b7-01a9-412d-a39d-ef94ea4c7103\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r" Oct 30 01:32:48.846823 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.846789 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba7596b7-01a9-412d-a39d-ef94ea4c7103-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r\" (UID: \"ba7596b7-01a9-412d-a39d-ef94ea4c7103\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r" Oct 30 01:32:48.846823 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.846806 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba7596b7-01a9-412d-a39d-ef94ea4c7103-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r\" (UID: \"ba7596b7-01a9-412d-a39d-ef94ea4c7103\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r" Oct 30 01:32:48.854292 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.854271 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 01:32:48.854805 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.854789 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r"] Oct 30 01:32:48.934828 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.934793 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt"] Oct 30 01:32:48.937318 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.937297 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" Oct 30 01:32:48.947753 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.947731 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11379e5f-0c93-49ac-9efa-c216820089ef-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt\" (UID: \"11379e5f-0c93-49ac-9efa-c216820089ef\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" Oct 30 01:32:48.947862 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.947763 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba7596b7-01a9-412d-a39d-ef94ea4c7103-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r\" (UID: \"ba7596b7-01a9-412d-a39d-ef94ea4c7103\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r" Oct 30 01:32:48.947862 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.947782 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba7596b7-01a9-412d-a39d-ef94ea4c7103-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r\" (UID: \"ba7596b7-01a9-412d-a39d-ef94ea4c7103\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r" Oct 30 01:32:48.947934 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.947883 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t796f\" (UniqueName: \"kubernetes.io/projected/11379e5f-0c93-49ac-9efa-c216820089ef-kube-api-access-t796f\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt\" (UID: \"11379e5f-0c93-49ac-9efa-c216820089ef\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" Oct 30 01:32:48.947988 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.947967 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v62h2\" (UniqueName: \"kubernetes.io/projected/ba7596b7-01a9-412d-a39d-ef94ea4c7103-kube-api-access-v62h2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r\" (UID: \"ba7596b7-01a9-412d-a39d-ef94ea4c7103\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r" Oct 30 01:32:48.948187 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.948169 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba7596b7-01a9-412d-a39d-ef94ea4c7103-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r\" (UID: \"ba7596b7-01a9-412d-a39d-ef94ea4c7103\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r" Oct 30 01:32:48.948241 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.948176 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11379e5f-0c93-49ac-9efa-c216820089ef-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt\" (UID: \"11379e5f-0c93-49ac-9efa-c216820089ef\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" Oct 30 01:32:48.948276 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.948253 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba7596b7-01a9-412d-a39d-ef94ea4c7103-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r\" (UID: \"ba7596b7-01a9-412d-a39d-ef94ea4c7103\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r" Oct 30 01:32:48.955789 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.955774 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-5r2bt\"" Oct 30 01:32:48.964459 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.964443 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt"] Oct 30 01:32:48.967469 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.967447 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:48.972403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:48.972380 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v62h2\" (UniqueName: \"kubernetes.io/projected/ba7596b7-01a9-412d-a39d-ef94ea4c7103-kube-api-access-v62h2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r\" (UID: \"ba7596b7-01a9-412d-a39d-ef94ea4c7103\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r" Oct 30 01:32:49.049103 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.049062 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11379e5f-0c93-49ac-9efa-c216820089ef-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt\" (UID: \"11379e5f-0c93-49ac-9efa-c216820089ef\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" Oct 30 01:32:49.049269 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.049128 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11379e5f-0c93-49ac-9efa-c216820089ef-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt\" (UID: \"11379e5f-0c93-49ac-9efa-c216820089ef\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" Oct 30 01:32:49.049269 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.049190 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t796f\" (UniqueName: \"kubernetes.io/projected/11379e5f-0c93-49ac-9efa-c216820089ef-kube-api-access-t796f\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt\" (UID: \"11379e5f-0c93-49ac-9efa-c216820089ef\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" Oct 30 01:32:49.049479 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.049449 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11379e5f-0c93-49ac-9efa-c216820089ef-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt\" (UID: \"11379e5f-0c93-49ac-9efa-c216820089ef\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" Oct 30 01:32:49.049546 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.049465 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11379e5f-0c93-49ac-9efa-c216820089ef-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt\" (UID: \"11379e5f-0c93-49ac-9efa-c216820089ef\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" Oct 30 01:32:49.070415 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.070389 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94"] Oct 30 01:32:49.072902 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.072882 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" Oct 30 01:32:49.081609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.081581 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t796f\" (UniqueName: \"kubernetes.io/projected/11379e5f-0c93-49ac-9efa-c216820089ef-kube-api-access-t796f\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt\" (UID: \"11379e5f-0c93-49ac-9efa-c216820089ef\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" Oct 30 01:32:49.089318 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.089295 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-hvdh8\"" Oct 30 01:32:49.097685 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.097664 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94"] Oct 30 01:32:49.138911 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.138875 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q"] Oct 30 01:32:49.139077 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.138975 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r" Oct 30 01:32:49.141214 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.141190 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" Oct 30 01:32:49.150326 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.150304 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e6221db-f57a-4d78-ae5f-7132742e99e0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q\" (UID: \"7e6221db-f57a-4d78-ae5f-7132742e99e0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" Oct 30 01:32:49.150413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.150351 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnrml\" (UniqueName: \"kubernetes.io/projected/7e6221db-f57a-4d78-ae5f-7132742e99e0-kube-api-access-mnrml\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q\" (UID: \"7e6221db-f57a-4d78-ae5f-7132742e99e0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" Oct 30 01:32:49.150413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.150381 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2ftr\" (UniqueName: \"kubernetes.io/projected/4a79b57b-f043-487c-a665-04bbc58a6443-kube-api-access-k2ftr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94\" (UID: \"4a79b57b-f043-487c-a665-04bbc58a6443\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" Oct 30 01:32:49.150504 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.150486 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a79b57b-f043-487c-a665-04bbc58a6443-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94\" (UID: \"4a79b57b-f043-487c-a665-04bbc58a6443\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" Oct 30 01:32:49.150539 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.150516 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a79b57b-f043-487c-a665-04bbc58a6443-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94\" (UID: \"4a79b57b-f043-487c-a665-04bbc58a6443\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" Oct 30 01:32:49.150572 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.150538 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e6221db-f57a-4d78-ae5f-7132742e99e0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q\" (UID: \"7e6221db-f57a-4d78-ae5f-7132742e99e0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" Oct 30 01:32:49.164142 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.164060 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q"] Oct 30 01:32:49.189799 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.189760 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-pdjzm\"" Oct 30 01:32:49.232488 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.232460 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr"] Oct 30 01:32:49.234777 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.234758 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr" Oct 30 01:32:49.249398 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.249378 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" Oct 30 01:32:49.250995 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.250929 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnrml\" (UniqueName: \"kubernetes.io/projected/7e6221db-f57a-4d78-ae5f-7132742e99e0-kube-api-access-mnrml\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q\" (UID: \"7e6221db-f57a-4d78-ae5f-7132742e99e0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" Oct 30 01:32:49.250995 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.250958 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2ftr\" (UniqueName: \"kubernetes.io/projected/4a79b57b-f043-487c-a665-04bbc58a6443-kube-api-access-k2ftr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94\" (UID: \"4a79b57b-f043-487c-a665-04bbc58a6443\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" Oct 30 01:32:49.250995 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.250979 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a79b57b-f043-487c-a665-04bbc58a6443-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94\" (UID: \"4a79b57b-f043-487c-a665-04bbc58a6443\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" Oct 30 01:32:49.251145 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.251004 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed147504-cb54-4593-8b39-526690e0a7ab-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr\" (UID: \"ed147504-cb54-4593-8b39-526690e0a7ab\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr" Oct 30 01:32:49.251145 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.251035 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a79b57b-f043-487c-a665-04bbc58a6443-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94\" (UID: \"4a79b57b-f043-487c-a665-04bbc58a6443\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" Oct 30 01:32:49.251145 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.251067 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e6221db-f57a-4d78-ae5f-7132742e99e0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q\" (UID: \"7e6221db-f57a-4d78-ae5f-7132742e99e0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" Oct 30 01:32:49.251145 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.251114 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed147504-cb54-4593-8b39-526690e0a7ab-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr\" (UID: \"ed147504-cb54-4593-8b39-526690e0a7ab\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr" Oct 30 01:32:49.251345 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.251215 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e6221db-f57a-4d78-ae5f-7132742e99e0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q\" (UID: \"7e6221db-f57a-4d78-ae5f-7132742e99e0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" Oct 30 01:32:49.251345 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.251263 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mc6d\" (UniqueName: \"kubernetes.io/projected/ed147504-cb54-4593-8b39-526690e0a7ab-kube-api-access-9mc6d\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr\" (UID: \"ed147504-cb54-4593-8b39-526690e0a7ab\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr" Oct 30 01:32:49.251424 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.251388 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a79b57b-f043-487c-a665-04bbc58a6443-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94\" (UID: \"4a79b57b-f043-487c-a665-04bbc58a6443\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" Oct 30 01:32:49.251466 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.251418 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a79b57b-f043-487c-a665-04bbc58a6443-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94\" (UID: \"4a79b57b-f043-487c-a665-04bbc58a6443\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" Oct 30 01:32:49.251504 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.251489 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e6221db-f57a-4d78-ae5f-7132742e99e0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q\" (UID: \"7e6221db-f57a-4d78-ae5f-7132742e99e0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" Oct 30 01:32:49.251547 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.251517 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e6221db-f57a-4d78-ae5f-7132742e99e0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q\" (UID: \"7e6221db-f57a-4d78-ae5f-7132742e99e0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" Oct 30 01:32:49.258794 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.258765 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr"] Oct 30 01:32:49.275035 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.275003 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnrml\" (UniqueName: \"kubernetes.io/projected/7e6221db-f57a-4d78-ae5f-7132742e99e0-kube-api-access-mnrml\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q\" (UID: \"7e6221db-f57a-4d78-ae5f-7132742e99e0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" Oct 30 01:32:49.275202 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.275008 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2ftr\" (UniqueName: \"kubernetes.io/projected/4a79b57b-f043-487c-a665-04bbc58a6443-kube-api-access-k2ftr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94\" (UID: \"4a79b57b-f043-487c-a665-04bbc58a6443\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" Oct 30 01:32:49.290952 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.290925 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-kb7b8\"" Oct 30 01:32:49.352519 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.352439 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed147504-cb54-4593-8b39-526690e0a7ab-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr\" (UID: \"ed147504-cb54-4593-8b39-526690e0a7ab\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr" Oct 30 01:32:49.352519 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.352501 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed147504-cb54-4593-8b39-526690e0a7ab-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr\" (UID: \"ed147504-cb54-4593-8b39-526690e0a7ab\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr" Oct 30 01:32:49.352686 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.352539 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mc6d\" (UniqueName: \"kubernetes.io/projected/ed147504-cb54-4593-8b39-526690e0a7ab-kube-api-access-9mc6d\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr\" (UID: \"ed147504-cb54-4593-8b39-526690e0a7ab\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr" Oct 30 01:32:49.352782 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.352762 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed147504-cb54-4593-8b39-526690e0a7ab-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr\" (UID: \"ed147504-cb54-4593-8b39-526690e0a7ab\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr" Oct 30 01:32:49.352868 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.352849 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed147504-cb54-4593-8b39-526690e0a7ab-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr\" (UID: \"ed147504-cb54-4593-8b39-526690e0a7ab\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr" Oct 30 01:32:49.375763 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.375734 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mc6d\" (UniqueName: \"kubernetes.io/projected/ed147504-cb54-4593-8b39-526690e0a7ab-kube-api-access-9mc6d\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr\" (UID: \"ed147504-cb54-4593-8b39-526690e0a7ab\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr" Oct 30 01:32:49.385585 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.385567 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" Oct 30 01:32:49.437401 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.437365 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r"] Oct 30 01:32:49.443951 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:32:49.443922 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7596b7_01a9_412d_a39d_ef94ea4c7103.slice/crio-fb2855f64226b3290f84484640f5994b91b2cbcd1c5cb14b54b00bd727b8f4e4 WatchSource:0}: Error finding container fb2855f64226b3290f84484640f5994b91b2cbcd1c5cb14b54b00bd727b8f4e4: Status 404 returned error can't find the container with id fb2855f64226b3290f84484640f5994b91b2cbcd1c5cb14b54b00bd727b8f4e4 Oct 30 01:32:49.445921 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.445904 5998 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:32:49.465759 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.465736 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" Oct 30 01:32:49.547122 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.547078 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt"] Oct 30 01:32:49.547271 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.547152 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr" Oct 30 01:32:49.554303 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:32:49.554271 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11379e5f_0c93_49ac_9efa_c216820089ef.slice/crio-c544c2fd8412bae26bf059c8428ad4646c03ded4e65008a6083382fda19451dd WatchSource:0}: Error finding container c544c2fd8412bae26bf059c8428ad4646c03ded4e65008a6083382fda19451dd: Status 404 returned error can't find the container with id c544c2fd8412bae26bf059c8428ad4646c03ded4e65008a6083382fda19451dd Oct 30 01:32:49.679918 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.679892 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94"] Oct 30 01:32:49.684730 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:32:49.684704 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a79b57b_f043_487c_a665_04bbc58a6443.slice/crio-e8049c045fed0d99ef5bbf02dbc042414c8fdb8f421feffa5128d145d33c7125 WatchSource:0}: Error finding container e8049c045fed0d99ef5bbf02dbc042414c8fdb8f421feffa5128d145d33c7125: Status 404 returned error can't find the container with id e8049c045fed0d99ef5bbf02dbc042414c8fdb8f421feffa5128d145d33c7125 Oct 30 01:32:49.760543 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.760508 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q"] Oct 30 01:32:49.765804 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:32:49.765778 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6221db_f57a_4d78_ae5f_7132742e99e0.slice/crio-899c10ce4a5ecc911a2a02c907ecabdd547131ad57e93d1618aede08c1b47ee0 WatchSource:0}: Error finding container 899c10ce4a5ecc911a2a02c907ecabdd547131ad57e93d1618aede08c1b47ee0: Status 404 returned error can't find the container with id 899c10ce4a5ecc911a2a02c907ecabdd547131ad57e93d1618aede08c1b47ee0 Oct 30 01:32:49.842298 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:49.842230 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr"] Oct 30 01:32:49.848495 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:32:49.848461 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded147504_cb54_4593_8b39_526690e0a7ab.slice/crio-4d1215cec144297a56910263019797ee270b2638bbbbc9e7d2c937a96a68f625 WatchSource:0}: Error finding container 4d1215cec144297a56910263019797ee270b2638bbbbc9e7d2c937a96a68f625: Status 404 returned error can't find the container with id 4d1215cec144297a56910263019797ee270b2638bbbbc9e7d2c937a96a68f625 Oct 30 01:32:50.250466 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.250426 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr" event={"ID":"ed147504-cb54-4593-8b39-526690e0a7ab","Type":"ContainerStarted","Data":"4d1215cec144297a56910263019797ee270b2638bbbbc9e7d2c937a96a68f625"} Oct 30 01:32:50.252826 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.252793 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" event={"ID":"4a79b57b-f043-487c-a665-04bbc58a6443","Type":"ContainerStarted","Data":"e8049c045fed0d99ef5bbf02dbc042414c8fdb8f421feffa5128d145d33c7125"} Oct 30 01:32:50.257260 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.257185 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" event={"ID":"7e6221db-f57a-4d78-ae5f-7132742e99e0","Type":"ContainerStarted","Data":"899c10ce4a5ecc911a2a02c907ecabdd547131ad57e93d1618aede08c1b47ee0"} Oct 30 01:32:50.259531 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.259459 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" event={"ID":"11379e5f-0c93-49ac-9efa-c216820089ef","Type":"ContainerStarted","Data":"c544c2fd8412bae26bf059c8428ad4646c03ded4e65008a6083382fda19451dd"} Oct 30 01:32:50.260850 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.260798 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r" event={"ID":"ba7596b7-01a9-412d-a39d-ef94ea4c7103","Type":"ContainerStarted","Data":"fb2855f64226b3290f84484640f5994b91b2cbcd1c5cb14b54b00bd727b8f4e4"} Oct 30 01:32:50.421747 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.421711 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5"] Oct 30 01:32:50.424980 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.424954 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5" Oct 30 01:32:50.439592 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.439564 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-7wmvp\"" Oct 30 01:32:50.453553 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.453469 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5"] Oct 30 01:32:50.462838 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.460959 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5\" (UID: \"ead8ca00-4c97-42b5-9fbf-cf5046ff726e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5" Oct 30 01:32:50.462838 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.461015 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nm28\" (UniqueName: \"kubernetes.io/projected/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-kube-api-access-8nm28\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5\" (UID: \"ead8ca00-4c97-42b5-9fbf-cf5046ff726e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5" Oct 30 01:32:50.462838 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.461048 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5\" (UID: \"ead8ca00-4c97-42b5-9fbf-cf5046ff726e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5" Oct 30 01:32:50.561552 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.561465 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5\" (UID: \"ead8ca00-4c97-42b5-9fbf-cf5046ff726e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5" Oct 30 01:32:50.561552 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.561525 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nm28\" (UniqueName: \"kubernetes.io/projected/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-kube-api-access-8nm28\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5\" (UID: \"ead8ca00-4c97-42b5-9fbf-cf5046ff726e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5" Oct 30 01:32:50.561552 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.561551 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5\" (UID: \"ead8ca00-4c97-42b5-9fbf-cf5046ff726e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5" Oct 30 01:32:50.561901 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.561867 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5\" (UID: \"ead8ca00-4c97-42b5-9fbf-cf5046ff726e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5" Oct 30 01:32:50.562049 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.562035 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5\" (UID: \"ead8ca00-4c97-42b5-9fbf-cf5046ff726e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5" Oct 30 01:32:50.586572 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.586528 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nm28\" (UniqueName: \"kubernetes.io/projected/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-kube-api-access-8nm28\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5\" (UID: \"ead8ca00-4c97-42b5-9fbf-cf5046ff726e\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5" Oct 30 01:32:50.742224 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:50.742185 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5" Oct 30 01:32:51.089305 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:51.089218 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5"] Oct 30 01:32:51.102906 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:32:51.102873 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead8ca00_4c97_42b5_9fbf_cf5046ff726e.slice/crio-043e6f7edc8ad1c3b77c235b54ee76a226fd4ef4320b4b6be6666c35e8a36471 WatchSource:0}: Error finding container 043e6f7edc8ad1c3b77c235b54ee76a226fd4ef4320b4b6be6666c35e8a36471: Status 404 returned error can't find the container with id 043e6f7edc8ad1c3b77c235b54ee76a226fd4ef4320b4b6be6666c35e8a36471 Oct 30 01:32:51.271497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:51.271392 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5" event={"ID":"ead8ca00-4c97-42b5-9fbf-cf5046ff726e","Type":"ContainerStarted","Data":"043e6f7edc8ad1c3b77c235b54ee76a226fd4ef4320b4b6be6666c35e8a36471"} Oct 30 01:32:51.601324 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:51.601283 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:51.742054 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:51.741975 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:52.723788 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.723749 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q"] Oct 30 01:32:52.727227 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.727199 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.743185 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.743155 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-9wm2v\"" Oct 30 01:32:52.743330 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.743225 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 01:32:52.755704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.755498 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 01:32:52.755704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.755578 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 01:32:52.765460 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.765432 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q"] Oct 30 01:32:52.782811 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.782780 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a8f11c-0b47-448d-9c4a-d98899aabf93-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.782949 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.782827 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0a8f11c-0b47-448d-9c4a-d98899aabf93-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.782949 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.782867 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a8f11c-0b47-448d-9c4a-d98899aabf93-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.782949 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.782892 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0a8f11c-0b47-448d-9c4a-d98899aabf93-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.782949 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.782922 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0a8f11c-0b47-448d-9c4a-d98899aabf93-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.783182 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.782972 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0a8f11c-0b47-448d-9c4a-d98899aabf93-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.783182 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.783014 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrzps\" (UniqueName: \"kubernetes.io/projected/b0a8f11c-0b47-448d-9c4a-d98899aabf93-kube-api-access-mrzps\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.783182 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.783078 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0a8f11c-0b47-448d-9c4a-d98899aabf93-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.783182 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.783126 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0a8f11c-0b47-448d-9c4a-d98899aabf93-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.811960 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.811909 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:52.884140 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.884106 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0a8f11c-0b47-448d-9c4a-d98899aabf93-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.884373 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.884168 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrzps\" (UniqueName: \"kubernetes.io/projected/b0a8f11c-0b47-448d-9c4a-d98899aabf93-kube-api-access-mrzps\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.884373 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.884219 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0a8f11c-0b47-448d-9c4a-d98899aabf93-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.884373 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.884249 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0a8f11c-0b47-448d-9c4a-d98899aabf93-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.884373 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.884298 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a8f11c-0b47-448d-9c4a-d98899aabf93-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.884373 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.884322 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0a8f11c-0b47-448d-9c4a-d98899aabf93-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.884373 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.884359 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a8f11c-0b47-448d-9c4a-d98899aabf93-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.884863 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.884381 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0a8f11c-0b47-448d-9c4a-d98899aabf93-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.884863 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.884411 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0a8f11c-0b47-448d-9c4a-d98899aabf93-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.884863 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.884465 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0a8f11c-0b47-448d-9c4a-d98899aabf93-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.884863 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.884650 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a8f11c-0b47-448d-9c4a-d98899aabf93-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.885163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.884989 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0a8f11c-0b47-448d-9c4a-d98899aabf93-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.885238 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.885216 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0a8f11c-0b47-448d-9c4a-d98899aabf93-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.885834 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.885796 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0a8f11c-0b47-448d-9c4a-d98899aabf93-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.888549 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.888486 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0a8f11c-0b47-448d-9c4a-d98899aabf93-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.891470 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.891448 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0a8f11c-0b47-448d-9c4a-d98899aabf93-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.933573 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.933540 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrzps\" (UniqueName: \"kubernetes.io/projected/b0a8f11c-0b47-448d-9c4a-d98899aabf93-kube-api-access-mrzps\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.935926 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.935896 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0a8f11c-0b47-448d-9c4a-d98899aabf93-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q\" (UID: \"b0a8f11c-0b47-448d-9c4a-d98899aabf93\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:52.983242 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:52.983109 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:53.047621 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:53.047336 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:32:53.101791 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:53.101754 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:53.213176 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:53.213109 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:53.400393 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:53.400310 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q"] Oct 30 01:32:53.408381 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:32:53.408284 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a8f11c_0b47_448d_9c4a_d98899aabf93.slice/crio-0636d5d4c852822ed05f1f21e4efb41e0b24ab956b9ccd5f3ac9aae0aa9d3581 WatchSource:0}: Error finding container 0636d5d4c852822ed05f1f21e4efb41e0b24ab956b9ccd5f3ac9aae0aa9d3581: Status 404 returned error can't find the container with id 0636d5d4c852822ed05f1f21e4efb41e0b24ab956b9ccd5f3ac9aae0aa9d3581 Oct 30 01:32:53.450452 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:53.448557 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h"] Oct 30 01:32:53.452516 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:53.452489 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" Oct 30 01:32:53.467894 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:53.467153 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-wl2r9\"" Oct 30 01:32:53.476915 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:53.476844 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h"] Oct 30 01:32:53.489719 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:53.489480 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6ts9\" (UniqueName: \"kubernetes.io/projected/1fd4361a-e33f-496d-84ca-14596544fcc1-kube-api-access-f6ts9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h\" (UID: \"1fd4361a-e33f-496d-84ca-14596544fcc1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" Oct 30 01:32:53.489719 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:53.489570 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fd4361a-e33f-496d-84ca-14596544fcc1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h\" (UID: \"1fd4361a-e33f-496d-84ca-14596544fcc1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" Oct 30 01:32:53.489719 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:53.489617 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fd4361a-e33f-496d-84ca-14596544fcc1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h\" (UID: \"1fd4361a-e33f-496d-84ca-14596544fcc1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" Oct 30 01:32:53.593461 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:53.590757 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fd4361a-e33f-496d-84ca-14596544fcc1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h\" (UID: \"1fd4361a-e33f-496d-84ca-14596544fcc1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" Oct 30 01:32:53.593461 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:53.590844 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fd4361a-e33f-496d-84ca-14596544fcc1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h\" (UID: \"1fd4361a-e33f-496d-84ca-14596544fcc1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" Oct 30 01:32:53.593461 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:53.590942 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ts9\" (UniqueName: \"kubernetes.io/projected/1fd4361a-e33f-496d-84ca-14596544fcc1-kube-api-access-f6ts9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h\" (UID: \"1fd4361a-e33f-496d-84ca-14596544fcc1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" Oct 30 01:32:53.593461 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:53.592161 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fd4361a-e33f-496d-84ca-14596544fcc1-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h\" (UID: \"1fd4361a-e33f-496d-84ca-14596544fcc1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" Oct 30 01:32:53.593461 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:53.592697 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fd4361a-e33f-496d-84ca-14596544fcc1-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h\" (UID: \"1fd4361a-e33f-496d-84ca-14596544fcc1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" Oct 30 01:32:53.619505 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:53.619471 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6ts9\" (UniqueName: \"kubernetes.io/projected/1fd4361a-e33f-496d-84ca-14596544fcc1-kube-api-access-f6ts9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h\" (UID: \"1fd4361a-e33f-496d-84ca-14596544fcc1\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" Oct 30 01:32:53.773360 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:53.773081 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" Oct 30 01:32:54.111936 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:54.111852 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h"] Oct 30 01:32:54.298665 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:54.297739 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" event={"ID":"1fd4361a-e33f-496d-84ca-14596544fcc1","Type":"ContainerStarted","Data":"8b1a949c7e02f6f4618ef064590279374fc92507a9ede2b32e5bf903fcf34694"} Oct 30 01:32:54.301887 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:54.301328 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" event={"ID":"b0a8f11c-0b47-448d-9c4a-d98899aabf93","Type":"ContainerStarted","Data":"0636d5d4c852822ed05f1f21e4efb41e0b24ab956b9ccd5f3ac9aae0aa9d3581"} Oct 30 01:32:55.279719 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:32:55.279664 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:32:56.253985 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:56.253913 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:32:56.682677 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:56.680373 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b"] Oct 30 01:32:56.685846 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:56.685163 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b" Oct 30 01:32:56.712218 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:56.712159 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b"] Oct 30 01:32:56.723124 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:56.722860 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38b4c088-eef0-4540-9736-a72358f0ffb3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b\" (UID: \"38b4c088-eef0-4540-9736-a72358f0ffb3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b" Oct 30 01:32:56.723124 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:56.722973 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38b4c088-eef0-4540-9736-a72358f0ffb3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b\" (UID: \"38b4c088-eef0-4540-9736-a72358f0ffb3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b" Oct 30 01:32:56.723124 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:56.723015 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrrn4\" (UniqueName: \"kubernetes.io/projected/38b4c088-eef0-4540-9736-a72358f0ffb3-kube-api-access-lrrn4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b\" (UID: \"38b4c088-eef0-4540-9736-a72358f0ffb3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b" Oct 30 01:32:56.824729 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:56.824110 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38b4c088-eef0-4540-9736-a72358f0ffb3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b\" (UID: \"38b4c088-eef0-4540-9736-a72358f0ffb3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b" Oct 30 01:32:56.824729 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:56.824660 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38b4c088-eef0-4540-9736-a72358f0ffb3-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b\" (UID: \"38b4c088-eef0-4540-9736-a72358f0ffb3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b" Oct 30 01:32:56.825181 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:56.824165 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrrn4\" (UniqueName: \"kubernetes.io/projected/38b4c088-eef0-4540-9736-a72358f0ffb3-kube-api-access-lrrn4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b\" (UID: \"38b4c088-eef0-4540-9736-a72358f0ffb3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b" Oct 30 01:32:56.825181 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:56.825085 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38b4c088-eef0-4540-9736-a72358f0ffb3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b\" (UID: \"38b4c088-eef0-4540-9736-a72358f0ffb3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b" Oct 30 01:32:56.825587 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:56.825566 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38b4c088-eef0-4540-9736-a72358f0ffb3-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b\" (UID: \"38b4c088-eef0-4540-9736-a72358f0ffb3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b" Oct 30 01:32:56.859647 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:56.859512 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrrn4\" (UniqueName: \"kubernetes.io/projected/38b4c088-eef0-4540-9736-a72358f0ffb3-kube-api-access-lrrn4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b\" (UID: \"38b4c088-eef0-4540-9736-a72358f0ffb3\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b" Oct 30 01:32:57.009400 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:57.008882 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b" Oct 30 01:32:57.396181 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:57.394991 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b"] Oct 30 01:32:58.338541 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:32:58.338452 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b" event={"ID":"38b4c088-eef0-4540-9736-a72358f0ffb3","Type":"ContainerStarted","Data":"16ec9d019229bf4d0e7952a92675f0dba663afa59f782177ca6ceb94ee013a38"} Oct 30 01:33:01.572778 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:01.572746 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:33:01.573202 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:01.572812 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:33:01.573202 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:01.572844 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:33:02.380539 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:02.380492 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" event={"ID":"b0a8f11c-0b47-448d-9c4a-d98899aabf93","Type":"ContainerStarted","Data":"6479030b8fc469a59aea361f9a037e070eaffb0f3b6241fb8a416121bca20797"} Oct 30 01:33:03.050241 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:03.048610 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:33:03.054793 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:03.054575 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:33:03.124564 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:03.123671 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" podStartSLOduration=2.964154688 podStartE2EDuration="11.12364992s" totalImagesPullingTime="8.159495232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:02.440029371 +0000 UTC m=+212.018820469" watchObservedRunningTime="2026-10-30 01:33:03.12364992 +0000 UTC m=+212.702440999" Oct 30 01:33:03.388018 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:03.387893 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:33:03.389762 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:03.389471 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q" Oct 30 01:33:05.400216 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:05.399527 5998 generic.go:356] "Generic (PLEG): container finished" podID="7e6221db-f57a-4d78-ae5f-7132742e99e0" containerID="8f1fee6c271315bbf71216bb7e559fd5f5c2a38e829e8d2f09ce5b37c54302cd" exitCode=0 Oct 30 01:33:05.400216 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:05.399837 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" event={"ID":"7e6221db-f57a-4d78-ae5f-7132742e99e0","Type":"ContainerDied","Data":"8f1fee6c271315bbf71216bb7e559fd5f5c2a38e829e8d2f09ce5b37c54302cd"} Oct 30 01:33:05.402037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:05.401736 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" event={"ID":"11379e5f-0c93-49ac-9efa-c216820089ef","Type":"ContainerStarted","Data":"71fccfd01d831bd87dd2fe203859ab61e0ca80353f8f2153dc4960b4340a0ffb"} Oct 30 01:33:05.403565 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:05.403538 5998 generic.go:356] "Generic (PLEG): container finished" podID="1fd4361a-e33f-496d-84ca-14596544fcc1" containerID="39a4c01df8de6d526f93a1f8c7e2758f790b54b15a5908d4bc0cdc72fca6e809" exitCode=0 Oct 30 01:33:05.403735 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:05.403669 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" event={"ID":"1fd4361a-e33f-496d-84ca-14596544fcc1","Type":"ContainerDied","Data":"39a4c01df8de6d526f93a1f8c7e2758f790b54b15a5908d4bc0cdc72fca6e809"} Oct 30 01:33:05.407264 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:05.407118 5998 generic.go:356] "Generic (PLEG): container finished" podID="ba7596b7-01a9-412d-a39d-ef94ea4c7103" containerID="85e637d49800fc63021eace9d119802e9ea8d231e32a9583e1469ef4cda3bdbb" exitCode=0 Oct 30 01:33:05.407264 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:05.407211 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r" event={"ID":"ba7596b7-01a9-412d-a39d-ef94ea4c7103","Type":"ContainerDied","Data":"85e637d49800fc63021eace9d119802e9ea8d231e32a9583e1469ef4cda3bdbb"} Oct 30 01:33:05.410303 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:05.410236 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" event={"ID":"4a79b57b-f043-487c-a665-04bbc58a6443","Type":"ContainerStarted","Data":"8306db6a538d1f4dba3b34f56eb317f48c57151c575d95cf20efca0cd34bd563"} Oct 30 01:33:06.417052 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:06.417018 5998 generic.go:356] "Generic (PLEG): container finished" podID="11379e5f-0c93-49ac-9efa-c216820089ef" containerID="71fccfd01d831bd87dd2fe203859ab61e0ca80353f8f2153dc4960b4340a0ffb" exitCode=0 Oct 30 01:33:06.417478 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:06.417145 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" event={"ID":"11379e5f-0c93-49ac-9efa-c216820089ef","Type":"ContainerDied","Data":"71fccfd01d831bd87dd2fe203859ab61e0ca80353f8f2153dc4960b4340a0ffb"} Oct 30 01:33:06.422114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:06.421696 5998 generic.go:356] "Generic (PLEG): container finished" podID="ead8ca00-4c97-42b5-9fbf-cf5046ff726e" containerID="a1b0a58dd75e530907117f383f44948fcd10beeec39b9b872df7cc56a57d7e70" exitCode=0 Oct 30 01:33:06.422114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:06.421786 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5" event={"ID":"ead8ca00-4c97-42b5-9fbf-cf5046ff726e","Type":"ContainerDied","Data":"a1b0a58dd75e530907117f383f44948fcd10beeec39b9b872df7cc56a57d7e70"} Oct 30 01:33:06.425112 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:06.424770 5998 generic.go:356] "Generic (PLEG): container finished" podID="ed147504-cb54-4593-8b39-526690e0a7ab" containerID="2547215fc1b0489b87f9642fb91195fd2e38bd70aa0e488b0758df838bb161c4" exitCode=0 Oct 30 01:33:06.425112 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:06.424993 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr" event={"ID":"ed147504-cb54-4593-8b39-526690e0a7ab","Type":"ContainerDied","Data":"2547215fc1b0489b87f9642fb91195fd2e38bd70aa0e488b0758df838bb161c4"} Oct 30 01:33:06.429399 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:06.429372 5998 generic.go:356] "Generic (PLEG): container finished" podID="38b4c088-eef0-4540-9736-a72358f0ffb3" containerID="50d2c1114e4c2bd5cc46f92df0e59e349d0c6d72e643d1fae1a1e0505d1bf644" exitCode=0 Oct 30 01:33:06.429504 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:06.429463 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b" event={"ID":"38b4c088-eef0-4540-9736-a72358f0ffb3","Type":"ContainerDied","Data":"50d2c1114e4c2bd5cc46f92df0e59e349d0c6d72e643d1fae1a1e0505d1bf644"} Oct 30 01:33:06.432036 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:06.432011 5998 generic.go:356] "Generic (PLEG): container finished" podID="4a79b57b-f043-487c-a665-04bbc58a6443" containerID="8306db6a538d1f4dba3b34f56eb317f48c57151c575d95cf20efca0cd34bd563" exitCode=0 Oct 30 01:33:06.432158 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:06.432135 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" event={"ID":"4a79b57b-f043-487c-a665-04bbc58a6443","Type":"ContainerDied","Data":"8306db6a538d1f4dba3b34f56eb317f48c57151c575d95cf20efca0cd34bd563"} Oct 30 01:33:07.439332 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:07.439290 5998 generic.go:356] "Generic (PLEG): container finished" podID="11379e5f-0c93-49ac-9efa-c216820089ef" containerID="fba215f74e911f2931e95a9bd87fa0a3ddd250edb2b7d5fd6828536623754b8f" exitCode=0 Oct 30 01:33:07.439738 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:07.439378 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" event={"ID":"11379e5f-0c93-49ac-9efa-c216820089ef","Type":"ContainerDied","Data":"fba215f74e911f2931e95a9bd87fa0a3ddd250edb2b7d5fd6828536623754b8f"} Oct 30 01:33:07.442142 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:07.442113 5998 generic.go:356] "Generic (PLEG): container finished" podID="ead8ca00-4c97-42b5-9fbf-cf5046ff726e" containerID="1a28dd8e53ac026002b571647777577f7dcece455ce6261b912e8d29c83ba7df" exitCode=0 Oct 30 01:33:07.442267 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:07.442201 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5" event={"ID":"ead8ca00-4c97-42b5-9fbf-cf5046ff726e","Type":"ContainerDied","Data":"1a28dd8e53ac026002b571647777577f7dcece455ce6261b912e8d29c83ba7df"} Oct 30 01:33:07.445417 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:07.445389 5998 generic.go:356] "Generic (PLEG): container finished" podID="ed147504-cb54-4593-8b39-526690e0a7ab" containerID="c393c9e32ffa0b9ffca5b21e3423bd7ff894f7541626bd1983f68ed76077265c" exitCode=0 Oct 30 01:33:07.445516 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:07.445457 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr" event={"ID":"ed147504-cb54-4593-8b39-526690e0a7ab","Type":"ContainerDied","Data":"c393c9e32ffa0b9ffca5b21e3423bd7ff894f7541626bd1983f68ed76077265c"} Oct 30 01:33:07.448407 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:07.448383 5998 generic.go:356] "Generic (PLEG): container finished" podID="ba7596b7-01a9-412d-a39d-ef94ea4c7103" containerID="1a30d47de063e9e4a570db5cfdf390dae8fb498b2bf0deda526e2e71d8ae5c31" exitCode=0 Oct 30 01:33:07.448634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:07.448609 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r" event={"ID":"ba7596b7-01a9-412d-a39d-ef94ea4c7103","Type":"ContainerDied","Data":"1a30d47de063e9e4a570db5cfdf390dae8fb498b2bf0deda526e2e71d8ae5c31"} Oct 30 01:33:07.451803 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:07.451779 5998 generic.go:356] "Generic (PLEG): container finished" podID="38b4c088-eef0-4540-9736-a72358f0ffb3" containerID="ec448127e70c7e1e9bb2e281a12659399fb2344bc07690d11410fa2ed03a410b" exitCode=0 Oct 30 01:33:07.451919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:07.451870 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b" event={"ID":"38b4c088-eef0-4540-9736-a72358f0ffb3","Type":"ContainerDied","Data":"ec448127e70c7e1e9bb2e281a12659399fb2344bc07690d11410fa2ed03a410b"} Oct 30 01:33:07.456025 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:07.456004 5998 generic.go:356] "Generic (PLEG): container finished" podID="4a79b57b-f043-487c-a665-04bbc58a6443" containerID="bb8e88d6302219bb786945a456b95e1c0c9c33d3890159a827ff4ab607874c70" exitCode=0 Oct 30 01:33:07.456120 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:07.456033 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" event={"ID":"4a79b57b-f043-487c-a665-04bbc58a6443","Type":"ContainerDied","Data":"bb8e88d6302219bb786945a456b95e1c0c9c33d3890159a827ff4ab607874c70"} Oct 30 01:33:08.471666 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:08.471626 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" event={"ID":"7e6221db-f57a-4d78-ae5f-7132742e99e0","Type":"ContainerStarted","Data":"17d04e1a4fdb3d6a8ed6729214c324bb47604a0e7e900b4a2df50344b72bda68"} Oct 30 01:33:08.478554 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:08.477809 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" event={"ID":"1fd4361a-e33f-496d-84ca-14596544fcc1","Type":"ContainerStarted","Data":"ec1090d22e561b170b8a3e5465c956f0ffb362971b83815668b4ab89ebe8bb36"} Oct 30 01:33:09.493959 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:09.493916 5998 generic.go:356] "Generic (PLEG): container finished" podID="1fd4361a-e33f-496d-84ca-14596544fcc1" containerID="ec1090d22e561b170b8a3e5465c956f0ffb362971b83815668b4ab89ebe8bb36" exitCode=0 Oct 30 01:33:09.494452 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:09.494109 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" event={"ID":"1fd4361a-e33f-496d-84ca-14596544fcc1","Type":"ContainerDied","Data":"ec1090d22e561b170b8a3e5465c956f0ffb362971b83815668b4ab89ebe8bb36"} Oct 30 01:33:09.509593 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:09.509554 5998 generic.go:356] "Generic (PLEG): container finished" podID="7e6221db-f57a-4d78-ae5f-7132742e99e0" containerID="17d04e1a4fdb3d6a8ed6729214c324bb47604a0e7e900b4a2df50344b72bda68" exitCode=0 Oct 30 01:33:09.509748 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:09.509616 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" event={"ID":"7e6221db-f57a-4d78-ae5f-7132742e99e0","Type":"ContainerDied","Data":"17d04e1a4fdb3d6a8ed6729214c324bb47604a0e7e900b4a2df50344b72bda68"} Oct 30 01:33:24.648348 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:24.648311 5998 generic.go:356] "Generic (PLEG): container finished" podID="11379e5f-0c93-49ac-9efa-c216820089ef" containerID="7931b2472dee6f8d7e09af56fc21ef7e9efa006724134a172532719f7274b1a4" exitCode=0 Oct 30 01:33:24.648935 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:24.648424 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" event={"ID":"11379e5f-0c93-49ac-9efa-c216820089ef","Type":"ContainerDied","Data":"7931b2472dee6f8d7e09af56fc21ef7e9efa006724134a172532719f7274b1a4"} Oct 30 01:33:25.215563 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:33:25.215530 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:33:25.655048 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:25.654967 5998 generic.go:356] "Generic (PLEG): container finished" podID="4a79b57b-f043-487c-a665-04bbc58a6443" containerID="d3f4307280322bc5fe1af8e7a1a6e6e299c93f44438e15bfaa4a15715a753f0c" exitCode=0 Oct 30 01:33:25.655472 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:25.655048 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" event={"ID":"4a79b57b-f043-487c-a665-04bbc58a6443","Type":"ContainerDied","Data":"d3f4307280322bc5fe1af8e7a1a6e6e299c93f44438e15bfaa4a15715a753f0c"} Oct 30 01:33:25.657187 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:25.657164 5998 generic.go:356] "Generic (PLEG): container finished" podID="ead8ca00-4c97-42b5-9fbf-cf5046ff726e" containerID="178c168a6dae5d5f34ea192636bcf65de51ded7ec94997f1a85f43e8799efae2" exitCode=0 Oct 30 01:33:25.657311 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:25.657191 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5" event={"ID":"ead8ca00-4c97-42b5-9fbf-cf5046ff726e","Type":"ContainerDied","Data":"178c168a6dae5d5f34ea192636bcf65de51ded7ec94997f1a85f43e8799efae2"} Oct 30 01:33:25.659146 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:25.659124 5998 generic.go:356] "Generic (PLEG): container finished" podID="1fd4361a-e33f-496d-84ca-14596544fcc1" containerID="01c36be03902142b90ece9533134cae01489dda4ea9e81cf6473324d74c14561" exitCode=0 Oct 30 01:33:25.659251 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:25.659186 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" event={"ID":"1fd4361a-e33f-496d-84ca-14596544fcc1","Type":"ContainerDied","Data":"01c36be03902142b90ece9533134cae01489dda4ea9e81cf6473324d74c14561"} Oct 30 01:33:25.661100 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:25.661066 5998 generic.go:356] "Generic (PLEG): container finished" podID="ed147504-cb54-4593-8b39-526690e0a7ab" containerID="e4de310c809a879a0f187022f684d49f15c3b17800514b4fb44c2ae8087a9007" exitCode=0 Oct 30 01:33:25.661200 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:25.661166 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr" event={"ID":"ed147504-cb54-4593-8b39-526690e0a7ab","Type":"ContainerDied","Data":"e4de310c809a879a0f187022f684d49f15c3b17800514b4fb44c2ae8087a9007"} Oct 30 01:33:25.663043 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:25.663024 5998 generic.go:356] "Generic (PLEG): container finished" podID="ba7596b7-01a9-412d-a39d-ef94ea4c7103" containerID="6a1ea9018d161111a8bc2710e17b4ad4383211a40cd79a5961f97c0b4ae6e925" exitCode=0 Oct 30 01:33:25.663165 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:25.663116 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r" event={"ID":"ba7596b7-01a9-412d-a39d-ef94ea4c7103","Type":"ContainerDied","Data":"6a1ea9018d161111a8bc2710e17b4ad4383211a40cd79a5961f97c0b4ae6e925"} Oct 30 01:33:25.825996 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:25.825975 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" Oct 30 01:33:25.941284 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:25.941250 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11379e5f-0c93-49ac-9efa-c216820089ef-kube-api-access-t796f\" (UniqueName: \"kubernetes.io/projected/11379e5f-0c93-49ac-9efa-c216820089ef-kube-api-access-t796f\") pod \"11379e5f-0c93-49ac-9efa-c216820089ef\" (UID: \"11379e5f-0c93-49ac-9efa-c216820089ef\") " Oct 30 01:33:25.941476 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:25.941338 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11379e5f-0c93-49ac-9efa-c216820089ef-bundle\" (UniqueName: \"kubernetes.io/empty-dir/11379e5f-0c93-49ac-9efa-c216820089ef-bundle\") pod \"11379e5f-0c93-49ac-9efa-c216820089ef\" (UID: \"11379e5f-0c93-49ac-9efa-c216820089ef\") " Oct 30 01:33:25.941476 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:25.941378 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11379e5f-0c93-49ac-9efa-c216820089ef-util\" (UniqueName: \"kubernetes.io/empty-dir/11379e5f-0c93-49ac-9efa-c216820089ef-util\") pod \"11379e5f-0c93-49ac-9efa-c216820089ef\" (UID: \"11379e5f-0c93-49ac-9efa-c216820089ef\") " Oct 30 01:33:25.941871 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:25.941846 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11379e5f-0c93-49ac-9efa-c216820089ef-bundle" pod "11379e5f-0c93-49ac-9efa-c216820089ef" (UID: "11379e5f-0c93-49ac-9efa-c216820089ef"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:25.943614 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:25.943589 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11379e5f-0c93-49ac-9efa-c216820089ef-kube-api-access-t796f" pod "11379e5f-0c93-49ac-9efa-c216820089ef" (UID: "11379e5f-0c93-49ac-9efa-c216820089ef"). InnerVolumeSpecName "kube-api-access-t796f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:25.947628 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:25.947597 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11379e5f-0c93-49ac-9efa-c216820089ef-util" pod "11379e5f-0c93-49ac-9efa-c216820089ef" (UID: "11379e5f-0c93-49ac-9efa-c216820089ef"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:26.042452 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.042420 5998 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11379e5f-0c93-49ac-9efa-c216820089ef-bundle\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:26.042452 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.042448 5998 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11379e5f-0c93-49ac-9efa-c216820089ef-util\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:26.042625 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.042457 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t796f\" (UniqueName: \"kubernetes.io/projected/11379e5f-0c93-49ac-9efa-c216820089ef-kube-api-access-t796f\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:26.669143 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.669107 5998 generic.go:356] "Generic (PLEG): container finished" podID="38b4c088-eef0-4540-9736-a72358f0ffb3" containerID="4bc1ce2008de1dae7112d8546ba78a7a5f403859914eafee02a71bd06e2b0da2" exitCode=0 Oct 30 01:33:26.669579 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.669188 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b" event={"ID":"38b4c088-eef0-4540-9736-a72358f0ffb3","Type":"ContainerDied","Data":"4bc1ce2008de1dae7112d8546ba78a7a5f403859914eafee02a71bd06e2b0da2"} Oct 30 01:33:26.671253 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.671223 5998 generic.go:356] "Generic (PLEG): container finished" podID="7e6221db-f57a-4d78-ae5f-7132742e99e0" containerID="d85aa3b8eff8dc1e6ab7ac1d8c093362e1685d961201f8f2782f60e231645c2b" exitCode=0 Oct 30 01:33:26.671372 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.671305 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" event={"ID":"7e6221db-f57a-4d78-ae5f-7132742e99e0","Type":"ContainerDied","Data":"d85aa3b8eff8dc1e6ab7ac1d8c093362e1685d961201f8f2782f60e231645c2b"} Oct 30 01:33:26.672979 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.672955 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" event={"ID":"11379e5f-0c93-49ac-9efa-c216820089ef","Type":"ContainerDied","Data":"c544c2fd8412bae26bf059c8428ad4646c03ded4e65008a6083382fda19451dd"} Oct 30 01:33:26.673119 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.672988 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c544c2fd8412bae26bf059c8428ad4646c03ded4e65008a6083382fda19451dd" Oct 30 01:33:26.673119 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.673047 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqsjbt" Oct 30 01:33:26.815643 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.815616 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5" Oct 30 01:33:26.870114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.869050 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r" Oct 30 01:33:26.890040 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.890011 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" Oct 30 01:33:26.897111 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.897067 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" Oct 30 01:33:26.900031 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.900013 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr" Oct 30 01:33:26.950541 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.950507 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-kube-api-access-8nm28\" (UniqueName: \"kubernetes.io/projected/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-kube-api-access-8nm28\") pod \"ead8ca00-4c97-42b5-9fbf-cf5046ff726e\" (UID: \"ead8ca00-4c97-42b5-9fbf-cf5046ff726e\") " Oct 30 01:33:26.950717 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.950560 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-util\" (UniqueName: \"kubernetes.io/empty-dir/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-util\") pod \"ead8ca00-4c97-42b5-9fbf-cf5046ff726e\" (UID: \"ead8ca00-4c97-42b5-9fbf-cf5046ff726e\") " Oct 30 01:33:26.950717 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.950609 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-bundle\") pod \"ead8ca00-4c97-42b5-9fbf-cf5046ff726e\" (UID: \"ead8ca00-4c97-42b5-9fbf-cf5046ff726e\") " Oct 30 01:33:26.951153 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.951131 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-bundle" pod "ead8ca00-4c97-42b5-9fbf-cf5046ff726e" (UID: "ead8ca00-4c97-42b5-9fbf-cf5046ff726e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:26.952602 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.952582 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-kube-api-access-8nm28" pod "ead8ca00-4c97-42b5-9fbf-cf5046ff726e" (UID: "ead8ca00-4c97-42b5-9fbf-cf5046ff726e"). InnerVolumeSpecName "kube-api-access-8nm28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:26.955851 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:26.955830 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-util" pod "ead8ca00-4c97-42b5-9fbf-cf5046ff726e" (UID: "ead8ca00-4c97-42b5-9fbf-cf5046ff726e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:27.052067 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.051985 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a79b57b-f043-487c-a665-04bbc58a6443-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a79b57b-f043-487c-a665-04bbc58a6443-bundle\") pod \"4a79b57b-f043-487c-a665-04bbc58a6443\" (UID: \"4a79b57b-f043-487c-a665-04bbc58a6443\") " Oct 30 01:33:27.052067 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.052051 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a79b57b-f043-487c-a665-04bbc58a6443-kube-api-access-k2ftr\" (UniqueName: \"kubernetes.io/projected/4a79b57b-f043-487c-a665-04bbc58a6443-kube-api-access-k2ftr\") pod \"4a79b57b-f043-487c-a665-04bbc58a6443\" (UID: \"4a79b57b-f043-487c-a665-04bbc58a6443\") " Oct 30 01:33:27.052281 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.052116 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7596b7-01a9-412d-a39d-ef94ea4c7103-kube-api-access-v62h2\" (UniqueName: \"kubernetes.io/projected/ba7596b7-01a9-412d-a39d-ef94ea4c7103-kube-api-access-v62h2\") pod \"ba7596b7-01a9-412d-a39d-ef94ea4c7103\" (UID: \"ba7596b7-01a9-412d-a39d-ef94ea4c7103\") " Oct 30 01:33:27.052281 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.052178 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed147504-cb54-4593-8b39-526690e0a7ab-kube-api-access-9mc6d\" (UniqueName: \"kubernetes.io/projected/ed147504-cb54-4593-8b39-526690e0a7ab-kube-api-access-9mc6d\") pod \"ed147504-cb54-4593-8b39-526690e0a7ab\" (UID: \"ed147504-cb54-4593-8b39-526690e0a7ab\") " Oct 30 01:33:27.052281 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.052219 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7596b7-01a9-412d-a39d-ef94ea4c7103-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba7596b7-01a9-412d-a39d-ef94ea4c7103-bundle\") pod \"ba7596b7-01a9-412d-a39d-ef94ea4c7103\" (UID: \"ba7596b7-01a9-412d-a39d-ef94ea4c7103\") " Oct 30 01:33:27.052281 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.052260 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a79b57b-f043-487c-a665-04bbc58a6443-util\" (UniqueName: \"kubernetes.io/empty-dir/4a79b57b-f043-487c-a665-04bbc58a6443-util\") pod \"4a79b57b-f043-487c-a665-04bbc58a6443\" (UID: \"4a79b57b-f043-487c-a665-04bbc58a6443\") " Oct 30 01:33:27.052551 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.052525 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7596b7-01a9-412d-a39d-ef94ea4c7103-util\" (UniqueName: \"kubernetes.io/empty-dir/ba7596b7-01a9-412d-a39d-ef94ea4c7103-util\") pod \"ba7596b7-01a9-412d-a39d-ef94ea4c7103\" (UID: \"ba7596b7-01a9-412d-a39d-ef94ea4c7103\") " Oct 30 01:33:27.052628 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.052584 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed147504-cb54-4593-8b39-526690e0a7ab-util\" (UniqueName: \"kubernetes.io/empty-dir/ed147504-cb54-4593-8b39-526690e0a7ab-util\") pod \"ed147504-cb54-4593-8b39-526690e0a7ab\" (UID: \"ed147504-cb54-4593-8b39-526690e0a7ab\") " Oct 30 01:33:27.052628 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.052622 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fd4361a-e33f-496d-84ca-14596544fcc1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fd4361a-e33f-496d-84ca-14596544fcc1-bundle\") pod \"1fd4361a-e33f-496d-84ca-14596544fcc1\" (UID: \"1fd4361a-e33f-496d-84ca-14596544fcc1\") " Oct 30 01:33:27.052721 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.052658 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd4361a-e33f-496d-84ca-14596544fcc1-kube-api-access-f6ts9\" (UniqueName: \"kubernetes.io/projected/1fd4361a-e33f-496d-84ca-14596544fcc1-kube-api-access-f6ts9\") pod \"1fd4361a-e33f-496d-84ca-14596544fcc1\" (UID: \"1fd4361a-e33f-496d-84ca-14596544fcc1\") " Oct 30 01:33:27.052721 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.052687 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fd4361a-e33f-496d-84ca-14596544fcc1-util\" (UniqueName: \"kubernetes.io/empty-dir/1fd4361a-e33f-496d-84ca-14596544fcc1-util\") pod \"1fd4361a-e33f-496d-84ca-14596544fcc1\" (UID: \"1fd4361a-e33f-496d-84ca-14596544fcc1\") " Oct 30 01:33:27.052826 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.052723 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed147504-cb54-4593-8b39-526690e0a7ab-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed147504-cb54-4593-8b39-526690e0a7ab-bundle\") pod \"ed147504-cb54-4593-8b39-526690e0a7ab\" (UID: \"ed147504-cb54-4593-8b39-526690e0a7ab\") " Oct 30 01:33:27.052921 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.052893 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a79b57b-f043-487c-a665-04bbc58a6443-bundle" pod "4a79b57b-f043-487c-a665-04bbc58a6443" (UID: "4a79b57b-f043-487c-a665-04bbc58a6443"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:27.053052 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.053033 5998 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-bundle\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.053138 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.053058 5998 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a79b57b-f043-487c-a665-04bbc58a6443-bundle\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.053138 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.053075 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nm28\" (UniqueName: \"kubernetes.io/projected/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-kube-api-access-8nm28\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.053138 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.053106 5998 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ead8ca00-4c97-42b5-9fbf-cf5046ff726e-util\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.059114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.053412 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7596b7-01a9-412d-a39d-ef94ea4c7103-bundle" pod "ba7596b7-01a9-412d-a39d-ef94ea4c7103" (UID: "ba7596b7-01a9-412d-a39d-ef94ea4c7103"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:27.059114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.053439 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed147504-cb54-4593-8b39-526690e0a7ab-bundle" pod "ed147504-cb54-4593-8b39-526690e0a7ab" (UID: "ed147504-cb54-4593-8b39-526690e0a7ab"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:27.059114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.053459 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd4361a-e33f-496d-84ca-14596544fcc1-bundle" pod "1fd4361a-e33f-496d-84ca-14596544fcc1" (UID: "1fd4361a-e33f-496d-84ca-14596544fcc1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:27.059114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.054749 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7596b7-01a9-412d-a39d-ef94ea4c7103-kube-api-access-v62h2" pod "ba7596b7-01a9-412d-a39d-ef94ea4c7103" (UID: "ba7596b7-01a9-412d-a39d-ef94ea4c7103"). InnerVolumeSpecName "kube-api-access-v62h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:27.059114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.054922 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed147504-cb54-4593-8b39-526690e0a7ab-kube-api-access-9mc6d" pod "ed147504-cb54-4593-8b39-526690e0a7ab" (UID: "ed147504-cb54-4593-8b39-526690e0a7ab"). InnerVolumeSpecName "kube-api-access-9mc6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:27.059114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.055883 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd4361a-e33f-496d-84ca-14596544fcc1-kube-api-access-f6ts9" pod "1fd4361a-e33f-496d-84ca-14596544fcc1" (UID: "1fd4361a-e33f-496d-84ca-14596544fcc1"). InnerVolumeSpecName "kube-api-access-f6ts9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:27.059114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.057084 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a79b57b-f043-487c-a665-04bbc58a6443-kube-api-access-k2ftr" pod "4a79b57b-f043-487c-a665-04bbc58a6443" (UID: "4a79b57b-f043-487c-a665-04bbc58a6443"). InnerVolumeSpecName "kube-api-access-k2ftr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:27.059114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.058448 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a79b57b-f043-487c-a665-04bbc58a6443-util" pod "4a79b57b-f043-487c-a665-04bbc58a6443" (UID: "4a79b57b-f043-487c-a665-04bbc58a6443"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:27.060337 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.060316 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fd4361a-e33f-496d-84ca-14596544fcc1-util" pod "1fd4361a-e33f-496d-84ca-14596544fcc1" (UID: "1fd4361a-e33f-496d-84ca-14596544fcc1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:27.060523 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.060496 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7596b7-01a9-412d-a39d-ef94ea4c7103-util" pod "ba7596b7-01a9-412d-a39d-ef94ea4c7103" (UID: "ba7596b7-01a9-412d-a39d-ef94ea4c7103"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:27.060883 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.060860 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed147504-cb54-4593-8b39-526690e0a7ab-util" pod "ed147504-cb54-4593-8b39-526690e0a7ab" (UID: "ed147504-cb54-4593-8b39-526690e0a7ab"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:27.154226 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.154172 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mc6d\" (UniqueName: \"kubernetes.io/projected/ed147504-cb54-4593-8b39-526690e0a7ab-kube-api-access-9mc6d\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.154226 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.154223 5998 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba7596b7-01a9-412d-a39d-ef94ea4c7103-bundle\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.154226 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.154238 5998 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a79b57b-f043-487c-a665-04bbc58a6443-util\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.154481 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.154249 5998 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba7596b7-01a9-412d-a39d-ef94ea4c7103-util\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.154481 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.154261 5998 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ed147504-cb54-4593-8b39-526690e0a7ab-util\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.154481 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.154272 5998 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fd4361a-e33f-496d-84ca-14596544fcc1-bundle\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.154481 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.154284 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6ts9\" (UniqueName: \"kubernetes.io/projected/1fd4361a-e33f-496d-84ca-14596544fcc1-kube-api-access-f6ts9\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.154481 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.154295 5998 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fd4361a-e33f-496d-84ca-14596544fcc1-util\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.154481 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.154306 5998 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ed147504-cb54-4593-8b39-526690e0a7ab-bundle\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.154481 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.154317 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2ftr\" (UniqueName: \"kubernetes.io/projected/4a79b57b-f043-487c-a665-04bbc58a6443-kube-api-access-k2ftr\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.154481 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.154329 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v62h2\" (UniqueName: \"kubernetes.io/projected/ba7596b7-01a9-412d-a39d-ef94ea4c7103-kube-api-access-v62h2\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.679010 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.678983 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r" Oct 30 01:33:27.679462 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.678981 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alqs2r" event={"ID":"ba7596b7-01a9-412d-a39d-ef94ea4c7103","Type":"ContainerDied","Data":"fb2855f64226b3290f84484640f5994b91b2cbcd1c5cb14b54b00bd727b8f4e4"} Oct 30 01:33:27.679462 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.679120 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb2855f64226b3290f84484640f5994b91b2cbcd1c5cb14b54b00bd727b8f4e4" Oct 30 01:33:27.680865 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.680839 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" Oct 30 01:33:27.680865 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.680858 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7c4h94" event={"ID":"4a79b57b-f043-487c-a665-04bbc58a6443","Type":"ContainerDied","Data":"e8049c045fed0d99ef5bbf02dbc042414c8fdb8f421feffa5128d145d33c7125"} Oct 30 01:33:27.681042 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.680893 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8049c045fed0d99ef5bbf02dbc042414c8fdb8f421feffa5128d145d33c7125" Oct 30 01:33:27.682946 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.682930 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5" Oct 30 01:33:27.682946 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.682940 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71cckd5" event={"ID":"ead8ca00-4c97-42b5-9fbf-cf5046ff726e","Type":"ContainerDied","Data":"043e6f7edc8ad1c3b77c235b54ee76a226fd4ef4320b4b6be6666c35e8a36471"} Oct 30 01:33:27.683112 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.682976 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="043e6f7edc8ad1c3b77c235b54ee76a226fd4ef4320b4b6be6666c35e8a36471" Oct 30 01:33:27.684940 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.684922 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" Oct 30 01:33:27.684940 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.684928 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644ss24h" event={"ID":"1fd4361a-e33f-496d-84ca-14596544fcc1","Type":"ContainerDied","Data":"8b1a949c7e02f6f4618ef064590279374fc92507a9ede2b32e5bf903fcf34694"} Oct 30 01:33:27.685121 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.684963 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b1a949c7e02f6f4618ef064590279374fc92507a9ede2b32e5bf903fcf34694" Oct 30 01:33:27.687262 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.687131 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr" event={"ID":"ed147504-cb54-4593-8b39-526690e0a7ab","Type":"ContainerDied","Data":"4d1215cec144297a56910263019797ee270b2638bbbbc9e7d2c937a96a68f625"} Oct 30 01:33:27.687262 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.687156 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68m4wr" Oct 30 01:33:27.687262 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.687172 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d1215cec144297a56910263019797ee270b2638bbbbc9e7d2c937a96a68f625" Oct 30 01:33:27.836757 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.836736 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" Oct 30 01:33:27.838384 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.838364 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b" Oct 30 01:33:27.861496 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.861472 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e6221db-f57a-4d78-ae5f-7132742e99e0-kube-api-access-mnrml\" (UniqueName: \"kubernetes.io/projected/7e6221db-f57a-4d78-ae5f-7132742e99e0-kube-api-access-mnrml\") pod \"7e6221db-f57a-4d78-ae5f-7132742e99e0\" (UID: \"7e6221db-f57a-4d78-ae5f-7132742e99e0\") " Oct 30 01:33:27.861602 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.861508 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38b4c088-eef0-4540-9736-a72358f0ffb3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/38b4c088-eef0-4540-9736-a72358f0ffb3-bundle\") pod \"38b4c088-eef0-4540-9736-a72358f0ffb3\" (UID: \"38b4c088-eef0-4540-9736-a72358f0ffb3\") " Oct 30 01:33:27.861602 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.861564 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38b4c088-eef0-4540-9736-a72358f0ffb3-util\" (UniqueName: \"kubernetes.io/empty-dir/38b4c088-eef0-4540-9736-a72358f0ffb3-util\") pod \"38b4c088-eef0-4540-9736-a72358f0ffb3\" (UID: \"38b4c088-eef0-4540-9736-a72358f0ffb3\") " Oct 30 01:33:27.861718 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.861610 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6221db-f57a-4d78-ae5f-7132742e99e0-util\" (UniqueName: \"kubernetes.io/empty-dir/7e6221db-f57a-4d78-ae5f-7132742e99e0-util\") pod \"7e6221db-f57a-4d78-ae5f-7132742e99e0\" (UID: \"7e6221db-f57a-4d78-ae5f-7132742e99e0\") " Oct 30 01:33:27.861718 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.861650 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b4c088-eef0-4540-9736-a72358f0ffb3-kube-api-access-lrrn4\" (UniqueName: \"kubernetes.io/projected/38b4c088-eef0-4540-9736-a72358f0ffb3-kube-api-access-lrrn4\") pod \"38b4c088-eef0-4540-9736-a72358f0ffb3\" (UID: \"38b4c088-eef0-4540-9736-a72358f0ffb3\") " Oct 30 01:33:27.861718 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.861698 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6221db-f57a-4d78-ae5f-7132742e99e0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e6221db-f57a-4d78-ae5f-7132742e99e0-bundle\") pod \"7e6221db-f57a-4d78-ae5f-7132742e99e0\" (UID: \"7e6221db-f57a-4d78-ae5f-7132742e99e0\") " Oct 30 01:33:27.862145 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.862120 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b4c088-eef0-4540-9736-a72358f0ffb3-bundle" pod "38b4c088-eef0-4540-9736-a72358f0ffb3" (UID: "38b4c088-eef0-4540-9736-a72358f0ffb3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:27.862529 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.862502 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6221db-f57a-4d78-ae5f-7132742e99e0-bundle" pod "7e6221db-f57a-4d78-ae5f-7132742e99e0" (UID: "7e6221db-f57a-4d78-ae5f-7132742e99e0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:27.863664 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.863643 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6221db-f57a-4d78-ae5f-7132742e99e0-kube-api-access-mnrml" pod "7e6221db-f57a-4d78-ae5f-7132742e99e0" (UID: "7e6221db-f57a-4d78-ae5f-7132742e99e0"). InnerVolumeSpecName "kube-api-access-mnrml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:27.863783 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.863768 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b4c088-eef0-4540-9736-a72358f0ffb3-kube-api-access-lrrn4" pod "38b4c088-eef0-4540-9736-a72358f0ffb3" (UID: "38b4c088-eef0-4540-9736-a72358f0ffb3"). InnerVolumeSpecName "kube-api-access-lrrn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:33:27.869687 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.869664 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6221db-f57a-4d78-ae5f-7132742e99e0-util" pod "7e6221db-f57a-4d78-ae5f-7132742e99e0" (UID: "7e6221db-f57a-4d78-ae5f-7132742e99e0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:27.869752 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.869737 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b4c088-eef0-4540-9736-a72358f0ffb3-util" pod "38b4c088-eef0-4540-9736-a72358f0ffb3" (UID: "38b4c088-eef0-4540-9736-a72358f0ffb3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:33:27.962903 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.962817 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnrml\" (UniqueName: \"kubernetes.io/projected/7e6221db-f57a-4d78-ae5f-7132742e99e0-kube-api-access-mnrml\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.962903 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.962847 5998 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38b4c088-eef0-4540-9736-a72358f0ffb3-bundle\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.962903 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.962857 5998 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38b4c088-eef0-4540-9736-a72358f0ffb3-util\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.962903 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.962865 5998 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7e6221db-f57a-4d78-ae5f-7132742e99e0-util\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.962903 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.962876 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrrn4\" (UniqueName: \"kubernetes.io/projected/38b4c088-eef0-4540-9736-a72358f0ffb3-kube-api-access-lrrn4\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:27.962903 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:27.962885 5998 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7e6221db-f57a-4d78-ae5f-7132742e99e0-bundle\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:33:28.692981 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:28.692953 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" Oct 30 01:33:28.692981 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:28.692973 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c55l9q" event={"ID":"7e6221db-f57a-4d78-ae5f-7132742e99e0","Type":"ContainerDied","Data":"899c10ce4a5ecc911a2a02c907ecabdd547131ad57e93d1618aede08c1b47ee0"} Oct 30 01:33:28.693505 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:28.693024 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="899c10ce4a5ecc911a2a02c907ecabdd547131ad57e93d1618aede08c1b47ee0" Oct 30 01:33:28.695053 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:28.695026 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b" event={"ID":"38b4c088-eef0-4540-9736-a72358f0ffb3","Type":"ContainerDied","Data":"16ec9d019229bf4d0e7952a92675f0dba663afa59f782177ca6ceb94ee013a38"} Oct 30 01:33:28.695229 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:28.695052 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7k96b" Oct 30 01:33:28.695229 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:28.695071 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16ec9d019229bf4d0e7952a92675f0dba663afa59f782177ca6ceb94ee013a38" Oct 30 01:33:28.785529 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:28.785497 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:29.657741 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.657708 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:29.821751 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.821719 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:29.925619 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925585 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z"] Oct 30 01:33:29.925891 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925878 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ed147504-cb54-4593-8b39-526690e0a7ab" containerName="pull" Oct 30 01:33:29.925933 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925892 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed147504-cb54-4593-8b39-526690e0a7ab" containerName="pull" Oct 30 01:33:29.925933 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925902 5998 cpu_manager.go:414] "containerMap: removing container" podUID="11379e5f-0c93-49ac-9efa-c216820089ef" containerName="pull" Oct 30 01:33:29.925933 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925907 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11379e5f-0c93-49ac-9efa-c216820089ef" containerName="pull" Oct 30 01:33:29.925933 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925914 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ed147504-cb54-4593-8b39-526690e0a7ab" containerName="util" Oct 30 01:33:29.925933 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925919 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed147504-cb54-4593-8b39-526690e0a7ab" containerName="util" Oct 30 01:33:29.925933 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925931 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ead8ca00-4c97-42b5-9fbf-cf5046ff726e" containerName="pull" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925938 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead8ca00-4c97-42b5-9fbf-cf5046ff726e" containerName="pull" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925948 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ed147504-cb54-4593-8b39-526690e0a7ab" containerName="extract" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925952 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed147504-cb54-4593-8b39-526690e0a7ab" containerName="extract" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925958 5998 cpu_manager.go:414] "containerMap: removing container" podUID="38b4c088-eef0-4540-9736-a72358f0ffb3" containerName="extract" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925963 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b4c088-eef0-4540-9736-a72358f0ffb3" containerName="extract" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925971 5998 cpu_manager.go:414] "containerMap: removing container" podUID="11379e5f-0c93-49ac-9efa-c216820089ef" containerName="util" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925975 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11379e5f-0c93-49ac-9efa-c216820089ef" containerName="util" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925980 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1fd4361a-e33f-496d-84ca-14596544fcc1" containerName="pull" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925984 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd4361a-e33f-496d-84ca-14596544fcc1" containerName="pull" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925991 5998 cpu_manager.go:414] "containerMap: removing container" podUID="4a79b57b-f043-487c-a665-04bbc58a6443" containerName="util" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.925995 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a79b57b-f043-487c-a665-04bbc58a6443" containerName="util" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926002 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ead8ca00-4c97-42b5-9fbf-cf5046ff726e" containerName="util" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926008 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead8ca00-4c97-42b5-9fbf-cf5046ff726e" containerName="util" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926017 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ead8ca00-4c97-42b5-9fbf-cf5046ff726e" containerName="extract" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926021 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead8ca00-4c97-42b5-9fbf-cf5046ff726e" containerName="extract" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926027 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1fd4361a-e33f-496d-84ca-14596544fcc1" containerName="extract" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926031 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd4361a-e33f-496d-84ca-14596544fcc1" containerName="extract" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926042 5998 cpu_manager.go:414] "containerMap: removing container" podUID="4a79b57b-f043-487c-a665-04bbc58a6443" containerName="pull" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926046 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a79b57b-f043-487c-a665-04bbc58a6443" containerName="pull" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926053 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ba7596b7-01a9-412d-a39d-ef94ea4c7103" containerName="util" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926057 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7596b7-01a9-412d-a39d-ef94ea4c7103" containerName="util" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926068 5998 cpu_manager.go:414] "containerMap: removing container" podUID="38b4c088-eef0-4540-9736-a72358f0ffb3" containerName="util" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926072 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b4c088-eef0-4540-9736-a72358f0ffb3" containerName="util" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926077 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7e6221db-f57a-4d78-ae5f-7132742e99e0" containerName="pull" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926082 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e6221db-f57a-4d78-ae5f-7132742e99e0" containerName="pull" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926104 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7e6221db-f57a-4d78-ae5f-7132742e99e0" containerName="extract" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926111 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e6221db-f57a-4d78-ae5f-7132742e99e0" containerName="extract" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926121 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1fd4361a-e33f-496d-84ca-14596544fcc1" containerName="util" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926125 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd4361a-e33f-496d-84ca-14596544fcc1" containerName="util" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926131 5998 cpu_manager.go:414] "containerMap: removing container" podUID="38b4c088-eef0-4540-9736-a72358f0ffb3" containerName="pull" Oct 30 01:33:29.926126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926136 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b4c088-eef0-4540-9736-a72358f0ffb3" containerName="pull" Oct 30 01:33:29.926990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926142 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ba7596b7-01a9-412d-a39d-ef94ea4c7103" containerName="extract" Oct 30 01:33:29.926990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926147 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7596b7-01a9-412d-a39d-ef94ea4c7103" containerName="extract" Oct 30 01:33:29.926990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926154 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ba7596b7-01a9-412d-a39d-ef94ea4c7103" containerName="pull" Oct 30 01:33:29.926990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926158 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7596b7-01a9-412d-a39d-ef94ea4c7103" containerName="pull" Oct 30 01:33:29.926990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926163 5998 cpu_manager.go:414] "containerMap: removing container" podUID="11379e5f-0c93-49ac-9efa-c216820089ef" containerName="extract" Oct 30 01:33:29.926990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926167 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11379e5f-0c93-49ac-9efa-c216820089ef" containerName="extract" Oct 30 01:33:29.926990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926175 5998 cpu_manager.go:414] "containerMap: removing container" podUID="4a79b57b-f043-487c-a665-04bbc58a6443" containerName="extract" Oct 30 01:33:29.926990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926180 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a79b57b-f043-487c-a665-04bbc58a6443" containerName="extract" Oct 30 01:33:29.926990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926190 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7e6221db-f57a-4d78-ae5f-7132742e99e0" containerName="util" Oct 30 01:33:29.926990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926196 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e6221db-f57a-4d78-ae5f-7132742e99e0" containerName="util" Oct 30 01:33:29.926990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926253 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead8ca00-4c97-42b5-9fbf-cf5046ff726e" containerName="extract" Oct 30 01:33:29.926990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926261 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd4361a-e33f-496d-84ca-14596544fcc1" containerName="extract" Oct 30 01:33:29.926990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926274 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e6221db-f57a-4d78-ae5f-7132742e99e0" containerName="extract" Oct 30 01:33:29.926990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926280 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7596b7-01a9-412d-a39d-ef94ea4c7103" containerName="extract" Oct 30 01:33:29.926990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926286 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="11379e5f-0c93-49ac-9efa-c216820089ef" containerName="extract" Oct 30 01:33:29.926990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926292 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a79b57b-f043-487c-a665-04bbc58a6443" containerName="extract" Oct 30 01:33:29.926990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926299 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed147504-cb54-4593-8b39-526690e0a7ab" containerName="extract" Oct 30 01:33:29.926990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.926304 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b4c088-eef0-4540-9736-a72358f0ffb3" containerName="extract" Oct 30 01:33:29.929029 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.929014 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:29.944321 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.944295 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 01:33:29.944596 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.944573 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-79ppx\"" Oct 30 01:33:29.955945 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.955923 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 01:33:29.956007 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.955947 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 01:33:29.956007 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.955981 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 01:33:29.956007 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.955987 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 01:33:29.961019 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.961001 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:29.975639 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.975612 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z"] Oct 30 01:33:29.978136 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.978111 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df56a0c8-cd2c-498f-b13e-2c788a2e402f-webhook-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-d4r4z\" (UID: \"df56a0c8-cd2c-498f-b13e-2c788a2e402f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:29.978236 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.978162 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/df56a0c8-cd2c-498f-b13e-2c788a2e402f-metrics-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-d4r4z\" (UID: \"df56a0c8-cd2c-498f-b13e-2c788a2e402f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:29.978284 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.978253 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df56a0c8-cd2c-498f-b13e-2c788a2e402f-tmp\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-d4r4z\" (UID: \"df56a0c8-cd2c-498f-b13e-2c788a2e402f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:29.978327 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.978293 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/df56a0c8-cd2c-498f-b13e-2c788a2e402f-manifests\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-d4r4z\" (UID: \"df56a0c8-cd2c-498f-b13e-2c788a2e402f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:29.978327 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.978315 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd7ld\" (UniqueName: \"kubernetes.io/projected/df56a0c8-cd2c-498f-b13e-2c788a2e402f-kube-api-access-rd7ld\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-d4r4z\" (UID: \"df56a0c8-cd2c-498f-b13e-2c788a2e402f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:29.978456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:29.978435 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/df56a0c8-cd2c-498f-b13e-2c788a2e402f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-d4r4z\" (UID: \"df56a0c8-cd2c-498f-b13e-2c788a2e402f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:30.079023 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.078988 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/df56a0c8-cd2c-498f-b13e-2c788a2e402f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-d4r4z\" (UID: \"df56a0c8-cd2c-498f-b13e-2c788a2e402f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:30.079236 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.079035 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df56a0c8-cd2c-498f-b13e-2c788a2e402f-webhook-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-d4r4z\" (UID: \"df56a0c8-cd2c-498f-b13e-2c788a2e402f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:30.079236 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.079070 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/df56a0c8-cd2c-498f-b13e-2c788a2e402f-metrics-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-d4r4z\" (UID: \"df56a0c8-cd2c-498f-b13e-2c788a2e402f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:30.079236 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.079123 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df56a0c8-cd2c-498f-b13e-2c788a2e402f-tmp\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-d4r4z\" (UID: \"df56a0c8-cd2c-498f-b13e-2c788a2e402f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:30.079236 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.079152 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/df56a0c8-cd2c-498f-b13e-2c788a2e402f-manifests\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-d4r4z\" (UID: \"df56a0c8-cd2c-498f-b13e-2c788a2e402f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:30.079236 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.079172 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd7ld\" (UniqueName: \"kubernetes.io/projected/df56a0c8-cd2c-498f-b13e-2c788a2e402f-kube-api-access-rd7ld\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-d4r4z\" (UID: \"df56a0c8-cd2c-498f-b13e-2c788a2e402f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:30.079649 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.079615 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/df56a0c8-cd2c-498f-b13e-2c788a2e402f-manifests\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-d4r4z\" (UID: \"df56a0c8-cd2c-498f-b13e-2c788a2e402f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:30.081513 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.081480 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df56a0c8-cd2c-498f-b13e-2c788a2e402f-tmp\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-d4r4z\" (UID: \"df56a0c8-cd2c-498f-b13e-2c788a2e402f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:30.081721 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.081702 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/df56a0c8-cd2c-498f-b13e-2c788a2e402f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-d4r4z\" (UID: \"df56a0c8-cd2c-498f-b13e-2c788a2e402f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:30.081763 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.081705 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/df56a0c8-cd2c-498f-b13e-2c788a2e402f-metrics-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-d4r4z\" (UID: \"df56a0c8-cd2c-498f-b13e-2c788a2e402f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:30.081797 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.081770 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/df56a0c8-cd2c-498f-b13e-2c788a2e402f-webhook-cert\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-d4r4z\" (UID: \"df56a0c8-cd2c-498f-b13e-2c788a2e402f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:30.104567 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.104541 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd7ld\" (UniqueName: \"kubernetes.io/projected/df56a0c8-cd2c-498f-b13e-2c788a2e402f-kube-api-access-rd7ld\") pod \"opendatahub-operator-controller-manager-66dfc65cbf-d4r4z\" (UID: \"df56a0c8-cd2c-498f-b13e-2c788a2e402f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:30.169063 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.169032 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:30.242279 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.242200 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:30.281422 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.281387 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:30.432487 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.432453 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:30.546947 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.546858 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:30.593813 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.593780 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z"] Oct 30 01:33:30.598017 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:33:30.597991 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf56a0c8_cd2c_498f_b13e_2c788a2e402f.slice/crio-340ee463851e536dd010da14a8aab490e44f7a0a8eeaa587339d8ebdad05c784 WatchSource:0}: Error finding container 340ee463851e536dd010da14a8aab490e44f7a0a8eeaa587339d8ebdad05c784: Status 404 returned error can't find the container with id 340ee463851e536dd010da14a8aab490e44f7a0a8eeaa587339d8ebdad05c784 Oct 30 01:33:30.709165 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:30.709121 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" event={"ID":"df56a0c8-cd2c-498f-b13e-2c788a2e402f","Type":"ContainerStarted","Data":"340ee463851e536dd010da14a8aab490e44f7a0a8eeaa587339d8ebdad05c784"} Oct 30 01:33:31.808885 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:31.808838 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:31.938506 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:31.938466 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:33.027253 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:33.027211 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:33.162282 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:33.162244 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:33.454382 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:33.454351 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-6kg7d"] Oct 30 01:33:33.457507 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:33.457485 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6kg7d" Oct 30 01:33:33.485938 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:33.485904 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:33:33.486212 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:33.486190 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 01:33:33.486330 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:33.486283 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-s778j\"" Oct 30 01:33:33.487553 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:33.487532 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-6kg7d"] Oct 30 01:33:33.507609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:33.507577 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed52726a-eb90-4a95-8e14-0877f8b29814-tmp\") pod \"jobset-operator-674cd59dc5-6kg7d\" (UID: \"ed52726a-eb90-4a95-8e14-0877f8b29814\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6kg7d" Oct 30 01:33:33.507767 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:33.507657 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gszn\" (UniqueName: \"kubernetes.io/projected/ed52726a-eb90-4a95-8e14-0877f8b29814-kube-api-access-9gszn\") pod \"jobset-operator-674cd59dc5-6kg7d\" (UID: \"ed52726a-eb90-4a95-8e14-0877f8b29814\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6kg7d" Oct 30 01:33:33.609017 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:33.608923 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gszn\" (UniqueName: \"kubernetes.io/projected/ed52726a-eb90-4a95-8e14-0877f8b29814-kube-api-access-9gszn\") pod \"jobset-operator-674cd59dc5-6kg7d\" (UID: \"ed52726a-eb90-4a95-8e14-0877f8b29814\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6kg7d" Oct 30 01:33:33.609232 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:33.609027 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed52726a-eb90-4a95-8e14-0877f8b29814-tmp\") pod \"jobset-operator-674cd59dc5-6kg7d\" (UID: \"ed52726a-eb90-4a95-8e14-0877f8b29814\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6kg7d" Oct 30 01:33:33.609454 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:33.609435 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed52726a-eb90-4a95-8e14-0877f8b29814-tmp\") pod \"jobset-operator-674cd59dc5-6kg7d\" (UID: \"ed52726a-eb90-4a95-8e14-0877f8b29814\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6kg7d" Oct 30 01:33:33.631907 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:33.631873 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gszn\" (UniqueName: \"kubernetes.io/projected/ed52726a-eb90-4a95-8e14-0877f8b29814-kube-api-access-9gszn\") pod \"jobset-operator-674cd59dc5-6kg7d\" (UID: \"ed52726a-eb90-4a95-8e14-0877f8b29814\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6kg7d" Oct 30 01:33:33.724584 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:33.724547 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" event={"ID":"df56a0c8-cd2c-498f-b13e-2c788a2e402f","Type":"ContainerStarted","Data":"989572240a25eacc76d235798f4199073acbf166785f8d7a3e9147329d55f674"} Oct 30 01:33:33.782601 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:33.782568 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6kg7d" Oct 30 01:33:34.097983 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:34.097914 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-6kg7d"] Oct 30 01:33:34.102843 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:33:34.102811 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded52726a_eb90_4a95_8e14_0877f8b29814.slice/crio-3edbfd3b895611012b344f899cdc4d7825ca8a6ec02c00e3b3397f2bf1a13447 WatchSource:0}: Error finding container 3edbfd3b895611012b344f899cdc4d7825ca8a6ec02c00e3b3397f2bf1a13447: Status 404 returned error can't find the container with id 3edbfd3b895611012b344f899cdc4d7825ca8a6ec02c00e3b3397f2bf1a13447 Oct 30 01:33:34.730429 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:34.730399 5998 generic.go:356] "Generic (PLEG): container finished" podID="df56a0c8-cd2c-498f-b13e-2c788a2e402f" containerID="989572240a25eacc76d235798f4199073acbf166785f8d7a3e9147329d55f674" exitCode=0 Oct 30 01:33:34.730634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:34.730496 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" event={"ID":"df56a0c8-cd2c-498f-b13e-2c788a2e402f","Type":"ContainerDied","Data":"989572240a25eacc76d235798f4199073acbf166785f8d7a3e9147329d55f674"} Oct 30 01:33:34.732106 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:34.731821 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6kg7d" event={"ID":"ed52726a-eb90-4a95-8e14-0877f8b29814","Type":"ContainerStarted","Data":"3edbfd3b895611012b344f899cdc4d7825ca8a6ec02c00e3b3397f2bf1a13447"} Oct 30 01:33:35.738985 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:35.738946 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" event={"ID":"df56a0c8-cd2c-498f-b13e-2c788a2e402f","Type":"ContainerStarted","Data":"c602564660ba4a94a281f10172a3efedd13d78fadca8528a72f2ce1560ca9106"} Oct 30 01:33:35.739500 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:35.739124 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:35.828988 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:35.828933 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" podStartSLOduration=3.35383111 podStartE2EDuration="6.828919124s" totalImagesPullingTime="3.041212936s" totalInitContainerRuntime="433.875078ms" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:35.81678839 +0000 UTC m=+245.395579472" watchObservedRunningTime="2026-10-30 01:33:35.828919124 +0000 UTC m=+245.407710206" Oct 30 01:33:36.745435 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:36.745401 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6kg7d" event={"ID":"ed52726a-eb90-4a95-8e14-0877f8b29814","Type":"ContainerStarted","Data":"e79e9b2a687a4168044fea7cd5994bf782eb70c6190fa43cad5c1f74b0c6ea71"} Oct 30 01:33:36.792660 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:36.792604 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-6kg7d" podStartSLOduration=1.89870361 podStartE2EDuration="3.792589863s" totalImagesPullingTime="1.893886253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:36.791662963 +0000 UTC m=+246.370454045" watchObservedRunningTime="2026-10-30 01:33:36.792589863 +0000 UTC m=+246.371380944" Oct 30 01:33:38.620279 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:38.620243 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:38.752996 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:38.752960 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:40.972192 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:40.972163 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:43.976680 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:43.976644 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:45.781784 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:45.781740 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:46.747962 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:46.747928 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-66dfc65cbf-d4r4z" Oct 30 01:33:47.945312 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:47.945262 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5"] Oct 30 01:33:47.951225 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:47.949325 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5" Oct 30 01:33:47.971045 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:47.971016 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:33:47.971045 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:47.971016 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 01:33:47.971227 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:47.971162 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-jdjp7\"" Oct 30 01:33:47.984135 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:47.984109 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5"] Oct 30 01:33:48.026268 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.026237 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed6fe376-0f83-4a1f-9b1d-384f79806da5-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5\" (UID: \"ed6fe376-0f83-4a1f-9b1d-384f79806da5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5" Oct 30 01:33:48.026420 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.026353 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxtcc\" (UniqueName: \"kubernetes.io/projected/ed6fe376-0f83-4a1f-9b1d-384f79806da5-kube-api-access-rxtcc\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5\" (UID: \"ed6fe376-0f83-4a1f-9b1d-384f79806da5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5" Oct 30 01:33:48.127659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.127622 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxtcc\" (UniqueName: \"kubernetes.io/projected/ed6fe376-0f83-4a1f-9b1d-384f79806da5-kube-api-access-rxtcc\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5\" (UID: \"ed6fe376-0f83-4a1f-9b1d-384f79806da5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5" Oct 30 01:33:48.127834 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.127681 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed6fe376-0f83-4a1f-9b1d-384f79806da5-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5\" (UID: \"ed6fe376-0f83-4a1f-9b1d-384f79806da5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5" Oct 30 01:33:48.128027 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.128010 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed6fe376-0f83-4a1f-9b1d-384f79806da5-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5\" (UID: \"ed6fe376-0f83-4a1f-9b1d-384f79806da5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5" Oct 30 01:33:48.152797 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.152760 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxtcc\" (UniqueName: \"kubernetes.io/projected/ed6fe376-0f83-4a1f-9b1d-384f79806da5-kube-api-access-rxtcc\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5\" (UID: \"ed6fe376-0f83-4a1f-9b1d-384f79806da5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5" Oct 30 01:33:48.262181 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.262102 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5" Oct 30 01:33:48.528455 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.528375 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-z4b6g"] Oct 30 01:33:48.530930 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.530906 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-z4b6g" Oct 30 01:33:48.556807 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.556784 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 01:33:48.557070 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.557056 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 01:33:48.560388 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.560370 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 01:33:48.560531 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.560519 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6h2nc\"" Oct 30 01:33:48.566284 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.566258 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-z4b6g"] Oct 30 01:33:48.603043 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.603008 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5"] Oct 30 01:33:48.615016 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:33:48.614990 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6fe376_0f83_4a1f_9b1d_384f79806da5.slice/crio-4ee4b2bcfe69807b582d0dbb728fe16c7add94d27fc2dbb63d3cb2d0e3554714 WatchSource:0}: Error finding container 4ee4b2bcfe69807b582d0dbb728fe16c7add94d27fc2dbb63d3cb2d0e3554714: Status 404 returned error can't find the container with id 4ee4b2bcfe69807b582d0dbb728fe16c7add94d27fc2dbb63d3cb2d0e3554714 Oct 30 01:33:48.632479 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.632451 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9rzk\" (UniqueName: \"kubernetes.io/projected/db689d73-bcb1-41fb-8507-84109c764878-kube-api-access-k9rzk\") pod \"dns-operator-controller-manager-8557cbdb84-z4b6g\" (UID: \"db689d73-bcb1-41fb-8507-84109c764878\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-z4b6g" Oct 30 01:33:48.733261 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.733220 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9rzk\" (UniqueName: \"kubernetes.io/projected/db689d73-bcb1-41fb-8507-84109c764878-kube-api-access-k9rzk\") pod \"dns-operator-controller-manager-8557cbdb84-z4b6g\" (UID: \"db689d73-bcb1-41fb-8507-84109c764878\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-z4b6g" Oct 30 01:33:48.769871 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.769841 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9rzk\" (UniqueName: \"kubernetes.io/projected/db689d73-bcb1-41fb-8507-84109c764878-kube-api-access-k9rzk\") pod \"dns-operator-controller-manager-8557cbdb84-z4b6g\" (UID: \"db689d73-bcb1-41fb-8507-84109c764878\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-z4b6g" Oct 30 01:33:48.797704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.797630 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5" event={"ID":"ed6fe376-0f83-4a1f-9b1d-384f79806da5","Type":"ContainerStarted","Data":"4ee4b2bcfe69807b582d0dbb728fe16c7add94d27fc2dbb63d3cb2d0e3554714"} Oct 30 01:33:48.842624 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.842587 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-z4b6g" Oct 30 01:33:48.853045 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:48.853018 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:49.159027 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:49.158879 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-z4b6g"] Oct 30 01:33:49.163855 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:33:49.163827 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb689d73_bcb1_41fb_8507_84109c764878.slice/crio-13db91c95ba02f8f6f9a42e003dc297e7632f6defabf4637018f1fe25b435677 WatchSource:0}: Error finding container 13db91c95ba02f8f6f9a42e003dc297e7632f6defabf4637018f1fe25b435677: Status 404 returned error can't find the container with id 13db91c95ba02f8f6f9a42e003dc297e7632f6defabf4637018f1fe25b435677 Oct 30 01:33:49.224218 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:49.224186 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:49.805841 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:49.805694 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-z4b6g" event={"ID":"db689d73-bcb1-41fb-8507-84109c764878","Type":"ContainerStarted","Data":"13db91c95ba02f8f6f9a42e003dc297e7632f6defabf4637018f1fe25b435677"} Oct 30 01:33:50.058214 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:50.058083 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:50.729358 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:50.729208 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:50.813414 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:50.813373 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5" event={"ID":"ed6fe376-0f83-4a1f-9b1d-384f79806da5","Type":"ContainerStarted","Data":"76a6ce55b70159ba2f09d73652cb834ca989cdf1d0023c59ca79d720355e1a22"} Oct 30 01:33:50.859126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:50.859075 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:50.881422 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:50.881363 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-s9fp5" podStartSLOduration=2.03156011 podStartE2EDuration="3.881345402s" totalImagesPullingTime="1.849785292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:50.869400894 +0000 UTC m=+260.448191976" watchObservedRunningTime="2026-10-30 01:33:50.881345402 +0000 UTC m=+260.460136482" Oct 30 01:33:51.176518 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:51.176483 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:51.819528 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:51.819490 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-z4b6g" event={"ID":"db689d73-bcb1-41fb-8507-84109c764878","Type":"ContainerStarted","Data":"52f5178f99d4f579eb3563fd06bff936e45fb14a7da4aae7f8c28f12cf74dc15"} Oct 30 01:33:51.901762 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:51.901702 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-z4b6g" podStartSLOduration=1.402460173 podStartE2EDuration="3.90168729s" totalImagesPullingTime="2.499227117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:51.900905373 +0000 UTC m=+261.479696466" watchObservedRunningTime="2026-10-30 01:33:51.90168729 +0000 UTC m=+261.480478370" Oct 30 01:33:52.642202 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:52.642168 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:52.823487 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:52.823459 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-z4b6g" Oct 30 01:33:52.878051 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:52.878018 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:53.047062 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:53.047023 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6"] Oct 30 01:33:53.068288 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:53.068265 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" Oct 30 01:33:53.073300 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:53.073218 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6"] Oct 30 01:33:53.087428 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:53.087365 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 01:33:53.093647 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:53.093628 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 01:33:53.114915 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:53.114890 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gsksl\"" Oct 30 01:33:53.171973 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:53.171942 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f02332d2-b193-4f97-bb47-d0d0904bdc9a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8pws6\" (UID: \"f02332d2-b193-4f97-bb47-d0d0904bdc9a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" Oct 30 01:33:53.172152 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:53.172013 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5sxr\" (UniqueName: \"kubernetes.io/projected/f02332d2-b193-4f97-bb47-d0d0904bdc9a-kube-api-access-j5sxr\") pod \"cert-manager-webhook-5dbbbf75cc-8pws6\" (UID: \"f02332d2-b193-4f97-bb47-d0d0904bdc9a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" Oct 30 01:33:53.273310 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:53.273252 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f02332d2-b193-4f97-bb47-d0d0904bdc9a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8pws6\" (UID: \"f02332d2-b193-4f97-bb47-d0d0904bdc9a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" Oct 30 01:33:53.273459 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:53.273346 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5sxr\" (UniqueName: \"kubernetes.io/projected/f02332d2-b193-4f97-bb47-d0d0904bdc9a-kube-api-access-j5sxr\") pod \"cert-manager-webhook-5dbbbf75cc-8pws6\" (UID: \"f02332d2-b193-4f97-bb47-d0d0904bdc9a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" Oct 30 01:33:53.300524 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:53.300455 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f02332d2-b193-4f97-bb47-d0d0904bdc9a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8pws6\" (UID: \"f02332d2-b193-4f97-bb47-d0d0904bdc9a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" Oct 30 01:33:53.300524 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:53.300494 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5sxr\" (UniqueName: \"kubernetes.io/projected/f02332d2-b193-4f97-bb47-d0d0904bdc9a-kube-api-access-j5sxr\") pod \"cert-manager-webhook-5dbbbf75cc-8pws6\" (UID: \"f02332d2-b193-4f97-bb47-d0d0904bdc9a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" Oct 30 01:33:53.396747 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:53.396711 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" Oct 30 01:33:53.720923 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:53.720883 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6"] Oct 30 01:33:53.726482 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:33:53.726455 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf02332d2_b193_4f97_bb47_d0d0904bdc9a.slice/crio-95bbd31513b643ad84f669bd4626a41e162d819868d592b09fde9f59aa57d8b3 WatchSource:0}: Error finding container 95bbd31513b643ad84f669bd4626a41e162d819868d592b09fde9f59aa57d8b3: Status 404 returned error can't find the container with id 95bbd31513b643ad84f669bd4626a41e162d819868d592b09fde9f59aa57d8b3 Oct 30 01:33:53.828813 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:53.828773 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" event={"ID":"f02332d2-b193-4f97-bb47-d0d0904bdc9a","Type":"ContainerStarted","Data":"95bbd31513b643ad84f669bd4626a41e162d819868d592b09fde9f59aa57d8b3"} Oct 30 01:33:55.160815 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:55.160777 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:55.220439 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:33:55.220398 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:33:55.399283 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:55.399241 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:55.532551 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:55.532484 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt"] Oct 30 01:33:55.535239 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:55.535218 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt" Oct 30 01:33:55.556393 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:55.556364 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v78hl\"" Oct 30 01:33:55.570324 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:55.570287 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt"] Oct 30 01:33:55.594402 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:55.594363 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72jkp\" (UniqueName: \"kubernetes.io/projected/994b558f-0da2-400b-af37-ac1a0a691fdc-kube-api-access-72jkp\") pod \"kuadrant-operator-controller-manager-678d87766f-w5pkt\" (UID: \"994b558f-0da2-400b-af37-ac1a0a691fdc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt" Oct 30 01:33:55.594588 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:55.594542 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/994b558f-0da2-400b-af37-ac1a0a691fdc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-w5pkt\" (UID: \"994b558f-0da2-400b-af37-ac1a0a691fdc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt" Oct 30 01:33:55.695505 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:55.695464 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/994b558f-0da2-400b-af37-ac1a0a691fdc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-w5pkt\" (UID: \"994b558f-0da2-400b-af37-ac1a0a691fdc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt" Oct 30 01:33:55.695687 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:55.695546 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72jkp\" (UniqueName: \"kubernetes.io/projected/994b558f-0da2-400b-af37-ac1a0a691fdc-kube-api-access-72jkp\") pod \"kuadrant-operator-controller-manager-678d87766f-w5pkt\" (UID: \"994b558f-0da2-400b-af37-ac1a0a691fdc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt" Oct 30 01:33:55.695865 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:55.695835 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/994b558f-0da2-400b-af37-ac1a0a691fdc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-w5pkt\" (UID: \"994b558f-0da2-400b-af37-ac1a0a691fdc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt" Oct 30 01:33:55.723629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:55.723594 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72jkp\" (UniqueName: \"kubernetes.io/projected/994b558f-0da2-400b-af37-ac1a0a691fdc-kube-api-access-72jkp\") pod \"kuadrant-operator-controller-manager-678d87766f-w5pkt\" (UID: \"994b558f-0da2-400b-af37-ac1a0a691fdc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt" Oct 30 01:33:55.849401 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:55.849125 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt" Oct 30 01:33:55.973713 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:55.973668 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:56.189328 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:56.189294 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt"] Oct 30 01:33:56.194633 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:33:56.194600 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994b558f_0da2_400b_af37_ac1a0a691fdc.slice/crio-1f53f3ee939455b1413e8d515a99563e05477f5acbf63e637cb57ea6adf40211 WatchSource:0}: Error finding container 1f53f3ee939455b1413e8d515a99563e05477f5acbf63e637cb57ea6adf40211: Status 404 returned error can't find the container with id 1f53f3ee939455b1413e8d515a99563e05477f5acbf63e637cb57ea6adf40211 Oct 30 01:33:56.784054 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:56.783977 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:56.847627 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:56.847580 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt" event={"ID":"994b558f-0da2-400b-af37-ac1a0a691fdc","Type":"ContainerStarted","Data":"1f53f3ee939455b1413e8d515a99563e05477f5acbf63e637cb57ea6adf40211"} Oct 30 01:33:56.849775 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:56.849743 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" event={"ID":"f02332d2-b193-4f97-bb47-d0d0904bdc9a","Type":"ContainerStarted","Data":"de6989d95a6a7b01cadc8e3926fcb5af52c35bac219cd3c6893ce062b1a91342"} Oct 30 01:33:56.849925 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:56.849901 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" Oct 30 01:33:56.913513 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:56.913450 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" podStartSLOduration=1.186146377 podStartE2EDuration="3.913429288s" totalImagesPullingTime="2.727282911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:56.90751756 +0000 UTC m=+266.486308641" watchObservedRunningTime="2026-10-30 01:33:56.913429288 +0000 UTC m=+266.492220371" Oct 30 01:33:59.195537 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:59.195495 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:59.382874 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:59.382835 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:59.532745 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:59.532713 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:33:59.870273 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:59.870151 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt" event={"ID":"994b558f-0da2-400b-af37-ac1a0a691fdc","Type":"ContainerStarted","Data":"cca2d07dbcdfa0da509f6acde4a9c8b139102732b6eaf9480b710a77dc301d01"} Oct 30 01:33:59.870473 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:59.870288 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt" Oct 30 01:33:59.937901 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:33:59.937842 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt" podStartSLOduration=1.653438264 podStartE2EDuration="4.937825968s" totalImagesPullingTime="3.284387704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:59.924630444 +0000 UTC m=+269.503421525" watchObservedRunningTime="2026-10-30 01:33:59.937825968 +0000 UTC m=+269.516617048" Oct 30 01:34:00.724737 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:00.724707 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:00.857920 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:00.857889 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:01.245560 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.245520 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:01.469844 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.469808 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:01.537521 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.537440 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7bd6fc9ff9-w5zwp"] Oct 30 01:34:01.540305 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.540284 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-w5zwp" Oct 30 01:34:01.577253 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.577226 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bd6fc9ff9-w5zwp"] Oct 30 01:34:01.641064 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.641028 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 01:34:01.641064 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.641034 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 01:34:01.641282 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.641117 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ljz6n\"" Oct 30 01:34:01.651458 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.651428 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/746a1452-89c9-4b38-9767-318905eaf8fe-tls-certs\") pod \"kube-auth-proxy-7bd6fc9ff9-w5zwp\" (UID: \"746a1452-89c9-4b38-9767-318905eaf8fe\") " pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-w5zwp" Oct 30 01:34:01.651596 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.651554 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/746a1452-89c9-4b38-9767-318905eaf8fe-tmp\") pod \"kube-auth-proxy-7bd6fc9ff9-w5zwp\" (UID: \"746a1452-89c9-4b38-9767-318905eaf8fe\") " pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-w5zwp" Oct 30 01:34:01.651645 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.651607 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlc6g\" (UniqueName: \"kubernetes.io/projected/746a1452-89c9-4b38-9767-318905eaf8fe-kube-api-access-tlc6g\") pod \"kube-auth-proxy-7bd6fc9ff9-w5zwp\" (UID: \"746a1452-89c9-4b38-9767-318905eaf8fe\") " pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-w5zwp" Oct 30 01:34:01.753017 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.752979 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/746a1452-89c9-4b38-9767-318905eaf8fe-tmp\") pod \"kube-auth-proxy-7bd6fc9ff9-w5zwp\" (UID: \"746a1452-89c9-4b38-9767-318905eaf8fe\") " pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-w5zwp" Oct 30 01:34:01.753436 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.753049 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlc6g\" (UniqueName: \"kubernetes.io/projected/746a1452-89c9-4b38-9767-318905eaf8fe-kube-api-access-tlc6g\") pod \"kube-auth-proxy-7bd6fc9ff9-w5zwp\" (UID: \"746a1452-89c9-4b38-9767-318905eaf8fe\") " pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-w5zwp" Oct 30 01:34:01.753436 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.753107 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/746a1452-89c9-4b38-9767-318905eaf8fe-tls-certs\") pod \"kube-auth-proxy-7bd6fc9ff9-w5zwp\" (UID: \"746a1452-89c9-4b38-9767-318905eaf8fe\") " pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-w5zwp" Oct 30 01:34:01.755288 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.755259 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/746a1452-89c9-4b38-9767-318905eaf8fe-tmp\") pod \"kube-auth-proxy-7bd6fc9ff9-w5zwp\" (UID: \"746a1452-89c9-4b38-9767-318905eaf8fe\") " pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-w5zwp" Oct 30 01:34:01.755560 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.755542 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/746a1452-89c9-4b38-9767-318905eaf8fe-tls-certs\") pod \"kube-auth-proxy-7bd6fc9ff9-w5zwp\" (UID: \"746a1452-89c9-4b38-9767-318905eaf8fe\") " pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-w5zwp" Oct 30 01:34:01.786000 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.785960 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlc6g\" (UniqueName: \"kubernetes.io/projected/746a1452-89c9-4b38-9767-318905eaf8fe-kube-api-access-tlc6g\") pod \"kube-auth-proxy-7bd6fc9ff9-w5zwp\" (UID: \"746a1452-89c9-4b38-9767-318905eaf8fe\") " pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-w5zwp" Oct 30 01:34:01.790901 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.790836 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:01.852984 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.852949 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-w5zwp" Oct 30 01:34:01.920780 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:01.920752 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:02.037953 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:02.037925 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:02.155573 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:02.155491 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bd6fc9ff9-w5zwp"] Oct 30 01:34:02.160252 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:34:02.160217 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746a1452_89c9_4b38_9767_318905eaf8fe.slice/crio-6d8da4550ff67fe96ef9d878a88b1c119abf8de53eb8d09ade6afb6859610b2f WatchSource:0}: Error finding container 6d8da4550ff67fe96ef9d878a88b1c119abf8de53eb8d09ade6afb6859610b2f: Status 404 returned error can't find the container with id 6d8da4550ff67fe96ef9d878a88b1c119abf8de53eb8d09ade6afb6859610b2f Oct 30 01:34:02.690291 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:02.690258 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:02.860311 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:02.860269 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:02.861638 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:02.861611 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" Oct 30 01:34:02.886669 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:02.886624 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-w5zwp" event={"ID":"746a1452-89c9-4b38-9767-318905eaf8fe","Type":"ContainerStarted","Data":"6d8da4550ff67fe96ef9d878a88b1c119abf8de53eb8d09ade6afb6859610b2f"} Oct 30 01:34:02.992792 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:02.992709 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:03.139336 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:03.139300 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:03.272323 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:03.272234 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:03.831183 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:03.831150 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-z4b6g" Oct 30 01:34:04.460991 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:04.460967 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:04.898337 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:04.898243 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-w5zwp" event={"ID":"746a1452-89c9-4b38-9767-318905eaf8fe","Type":"ContainerStarted","Data":"ca6b4a0c0aa08328dbeb6a028b732bc1bc19e73744547409423d39a1fbbefebe"} Oct 30 01:34:04.938771 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:04.938715 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7bd6fc9ff9-w5zwp" podStartSLOduration=1.642553876 podStartE2EDuration="3.938699997s" totalImagesPullingTime="2.296146121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:04.937705781 +0000 UTC m=+274.516496862" watchObservedRunningTime="2026-10-30 01:34:04.938699997 +0000 UTC m=+274.517491078" Oct 30 01:34:05.066229 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:05.066196 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:05.296603 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:05.296564 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:06.382272 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:06.382239 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:06.493134 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:06.493080 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:10.374165 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.374133 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4"] Oct 30 01:34:10.380451 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.380427 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" Oct 30 01:34:10.394722 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.394696 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-2drqg\"" Oct 30 01:34:10.406223 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.406199 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 01:34:10.406223 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.406196 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 01:34:10.430243 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.430206 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 01:34:10.431688 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.431666 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4"] Oct 30 01:34:10.530930 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.530894 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/114ffe72-9ce9-4eed-baea-d3667e969148-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-zr7w4\" (UID: \"114ffe72-9ce9-4eed-baea-d3667e969148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" Oct 30 01:34:10.531111 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.530937 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/114ffe72-9ce9-4eed-baea-d3667e969148-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-zr7w4\" (UID: \"114ffe72-9ce9-4eed-baea-d3667e969148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" Oct 30 01:34:10.531111 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.530963 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/114ffe72-9ce9-4eed-baea-d3667e969148-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-zr7w4\" (UID: \"114ffe72-9ce9-4eed-baea-d3667e969148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" Oct 30 01:34:10.531189 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.531161 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxvmf\" (UniqueName: \"kubernetes.io/projected/114ffe72-9ce9-4eed-baea-d3667e969148-kube-api-access-rxvmf\") pod \"kserve-module-controller-manager-7566cf5fc8-zr7w4\" (UID: \"114ffe72-9ce9-4eed-baea-d3667e969148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" Oct 30 01:34:10.631972 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.631879 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/114ffe72-9ce9-4eed-baea-d3667e969148-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-zr7w4\" (UID: \"114ffe72-9ce9-4eed-baea-d3667e969148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" Oct 30 01:34:10.631972 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.631919 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/114ffe72-9ce9-4eed-baea-d3667e969148-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-zr7w4\" (UID: \"114ffe72-9ce9-4eed-baea-d3667e969148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" Oct 30 01:34:10.631972 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.631969 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxvmf\" (UniqueName: \"kubernetes.io/projected/114ffe72-9ce9-4eed-baea-d3667e969148-kube-api-access-rxvmf\") pod \"kserve-module-controller-manager-7566cf5fc8-zr7w4\" (UID: \"114ffe72-9ce9-4eed-baea-d3667e969148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" Oct 30 01:34:10.632309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.632009 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/114ffe72-9ce9-4eed-baea-d3667e969148-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-zr7w4\" (UID: \"114ffe72-9ce9-4eed-baea-d3667e969148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" Oct 30 01:34:10.632309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.632290 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/114ffe72-9ce9-4eed-baea-d3667e969148-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-zr7w4\" (UID: \"114ffe72-9ce9-4eed-baea-d3667e969148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" Oct 30 01:34:10.632425 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.632319 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/114ffe72-9ce9-4eed-baea-d3667e969148-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-zr7w4\" (UID: \"114ffe72-9ce9-4eed-baea-d3667e969148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" Oct 30 01:34:10.632846 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.632819 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/114ffe72-9ce9-4eed-baea-d3667e969148-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-zr7w4\" (UID: \"114ffe72-9ce9-4eed-baea-d3667e969148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" Oct 30 01:34:10.655209 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.655177 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxvmf\" (UniqueName: \"kubernetes.io/projected/114ffe72-9ce9-4eed-baea-d3667e969148-kube-api-access-rxvmf\") pod \"kserve-module-controller-manager-7566cf5fc8-zr7w4\" (UID: \"114ffe72-9ce9-4eed-baea-d3667e969148\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" Oct 30 01:34:10.692017 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.691989 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" Oct 30 01:34:10.877624 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:10.877596 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt" Oct 30 01:34:11.009844 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:11.009805 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4"] Oct 30 01:34:11.665757 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:11.665719 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:11.929987 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:11.929900 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:11.941209 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:11.941169 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" event={"ID":"114ffe72-9ce9-4eed-baea-d3667e969148","Type":"ContainerStarted","Data":"ea243268cb70bbf94d02ca8fe04ea4fa6f23148932159af97bbee17ae78d6547"} Oct 30 01:34:12.076226 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:12.076183 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:12.947707 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:12.947670 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" event={"ID":"114ffe72-9ce9-4eed-baea-d3667e969148","Type":"ContainerStarted","Data":"7b518cd042431bf6c039082c47e3a8beaf23f89b274f7880f80859c535c7f580"} Oct 30 01:34:12.948161 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:12.947798 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" Oct 30 01:34:13.006308 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:13.006255 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" podStartSLOduration=1.536704278 podStartE2EDuration="3.006242215s" totalImagesPullingTime="1.469537937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:13.005319492 +0000 UTC m=+282.584110574" watchObservedRunningTime="2026-10-30 01:34:13.006242215 +0000 UTC m=+282.585033295" Oct 30 01:34:20.424770 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:20.424733 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:20.534764 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:20.534729 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:20.640798 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:20.640763 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:23.031976 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:23.031943 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:23.161862 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:23.161829 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:25.215164 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:34:25.215117 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:34:25.625336 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:25.625255 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:25.736862 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:25.736823 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:29.409656 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:29.409617 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:29.568628 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:29.568586 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:30.643258 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:30.643229 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:30.796316 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:30.796278 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:30.908730 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:30.908642 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:31.032490 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:31.032450 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:31.139362 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:31.139327 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:31.929790 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:31.929758 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:33.954058 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:33.954024 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-zr7w4" Oct 30 01:34:39.392003 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:39.391971 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:39.496218 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:34:39.496186 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:34:55.215809 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:34:55.215766 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:35:05.917117 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:05.917071 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:06.030441 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.030396 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:06.222259 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.222171 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:06.281744 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.281710 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-25r9k"] Oct 30 01:35:06.289680 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.289643 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-25r9k" Oct 30 01:35:06.308190 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.308162 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-25r9k"] Oct 30 01:35:06.318730 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.318699 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vchpz\"" Oct 30 01:35:06.318943 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.318841 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 01:35:06.334154 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.334123 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b"] Oct 30 01:35:06.337909 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.337888 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b" Oct 30 01:35:06.353450 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.353386 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:06.362482 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.362450 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b"] Oct 30 01:35:06.374750 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.374721 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-9fjjh"] Oct 30 01:35:06.378504 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.378482 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-9fjjh" Oct 30 01:35:06.402543 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.402502 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7994cb01-5d42-4871-ae17-d9622123f971-cert\") pod \"kserve-controller-manager-776cc66d47-25r9k\" (UID: \"7994cb01-5d42-4871-ae17-d9622123f971\") " pod="opendatahub/kserve-controller-manager-776cc66d47-25r9k" Oct 30 01:35:06.402736 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.402563 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5smd\" (UniqueName: \"kubernetes.io/projected/7994cb01-5d42-4871-ae17-d9622123f971-kube-api-access-s5smd\") pod \"kserve-controller-manager-776cc66d47-25r9k\" (UID: \"7994cb01-5d42-4871-ae17-d9622123f971\") " pod="opendatahub/kserve-controller-manager-776cc66d47-25r9k" Oct 30 01:35:06.403711 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.403683 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-9fjjh"] Oct 30 01:35:06.409825 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.409786 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2fwwb\"" Oct 30 01:35:06.409968 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.409832 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 01:35:06.413628 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.413601 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-4s5p5\"" Oct 30 01:35:06.413628 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.413608 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 01:35:06.432234 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.432201 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-kpkmz"] Oct 30 01:35:06.435682 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.435656 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-kpkmz" Oct 30 01:35:06.459601 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.459563 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-kpkmz"] Oct 30 01:35:06.503949 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.503856 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5smd\" (UniqueName: \"kubernetes.io/projected/7994cb01-5d42-4871-ae17-d9622123f971-kube-api-access-s5smd\") pod \"kserve-controller-manager-776cc66d47-25r9k\" (UID: \"7994cb01-5d42-4871-ae17-d9622123f971\") " pod="opendatahub/kserve-controller-manager-776cc66d47-25r9k" Oct 30 01:35:06.503949 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.503906 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v96t6\" (UniqueName: \"kubernetes.io/projected/27395a7d-e510-484a-b2ec-9863321c7c4b-kube-api-access-v96t6\") pod \"llmisvc-controller-manager-869d5dcc5d-w7p7b\" (UID: \"27395a7d-e510-484a-b2ec-9863321c7c4b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b" Oct 30 01:35:06.504170 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.504024 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmr2r\" (UniqueName: \"kubernetes.io/projected/712fc2f5-d537-4e96-9801-963522329de7-kube-api-access-kmr2r\") pod \"model-serving-api-75bd89fdfc-9fjjh\" (UID: \"712fc2f5-d537-4e96-9801-963522329de7\") " pod="opendatahub/model-serving-api-75bd89fdfc-9fjjh" Oct 30 01:35:06.504207 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.504179 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712fc2f5-d537-4e96-9801-963522329de7-tls-certs\") pod \"model-serving-api-75bd89fdfc-9fjjh\" (UID: \"712fc2f5-d537-4e96-9801-963522329de7\") " pod="opendatahub/model-serving-api-75bd89fdfc-9fjjh" Oct 30 01:35:06.504247 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.504213 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27395a7d-e510-484a-b2ec-9863321c7c4b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-w7p7b\" (UID: \"27395a7d-e510-484a-b2ec-9863321c7c4b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b" Oct 30 01:35:06.504316 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.504297 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7994cb01-5d42-4871-ae17-d9622123f971-cert\") pod \"kserve-controller-manager-776cc66d47-25r9k\" (UID: \"7994cb01-5d42-4871-ae17-d9622123f971\") " pod="opendatahub/kserve-controller-manager-776cc66d47-25r9k" Oct 30 01:35:06.506761 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.506732 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7994cb01-5d42-4871-ae17-d9622123f971-cert\") pod \"kserve-controller-manager-776cc66d47-25r9k\" (UID: \"7994cb01-5d42-4871-ae17-d9622123f971\") " pod="opendatahub/kserve-controller-manager-776cc66d47-25r9k" Oct 30 01:35:06.509672 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.509652 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 01:35:06.509672 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.509659 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-bb2hx\"" Oct 30 01:35:06.509778 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.509646 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 01:35:06.526749 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.526714 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5smd\" (UniqueName: \"kubernetes.io/projected/7994cb01-5d42-4871-ae17-d9622123f971-kube-api-access-s5smd\") pod \"kserve-controller-manager-776cc66d47-25r9k\" (UID: \"7994cb01-5d42-4871-ae17-d9622123f971\") " pod="opendatahub/kserve-controller-manager-776cc66d47-25r9k" Oct 30 01:35:06.602244 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.602211 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-25r9k" Oct 30 01:35:06.605288 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.605257 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v96t6\" (UniqueName: \"kubernetes.io/projected/27395a7d-e510-484a-b2ec-9863321c7c4b-kube-api-access-v96t6\") pod \"llmisvc-controller-manager-869d5dcc5d-w7p7b\" (UID: \"27395a7d-e510-484a-b2ec-9863321c7c4b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b" Oct 30 01:35:06.605862 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.605296 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmr2r\" (UniqueName: \"kubernetes.io/projected/712fc2f5-d537-4e96-9801-963522329de7-kube-api-access-kmr2r\") pod \"model-serving-api-75bd89fdfc-9fjjh\" (UID: \"712fc2f5-d537-4e96-9801-963522329de7\") " pod="opendatahub/model-serving-api-75bd89fdfc-9fjjh" Oct 30 01:35:06.605862 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.605317 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04727d53-1a8f-4f08-9930-acfb77f05eb4-cert\") pod \"odh-model-controller-769875db56-kpkmz\" (UID: \"04727d53-1a8f-4f08-9930-acfb77f05eb4\") " pod="opendatahub/odh-model-controller-769875db56-kpkmz" Oct 30 01:35:06.605862 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.605349 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712fc2f5-d537-4e96-9801-963522329de7-tls-certs\") pod \"model-serving-api-75bd89fdfc-9fjjh\" (UID: \"712fc2f5-d537-4e96-9801-963522329de7\") " pod="opendatahub/model-serving-api-75bd89fdfc-9fjjh" Oct 30 01:35:06.605862 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.605374 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27395a7d-e510-484a-b2ec-9863321c7c4b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-w7p7b\" (UID: \"27395a7d-e510-484a-b2ec-9863321c7c4b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b" Oct 30 01:35:06.605862 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.605406 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzbvw\" (UniqueName: \"kubernetes.io/projected/04727d53-1a8f-4f08-9930-acfb77f05eb4-kube-api-access-qzbvw\") pod \"odh-model-controller-769875db56-kpkmz\" (UID: \"04727d53-1a8f-4f08-9930-acfb77f05eb4\") " pod="opendatahub/odh-model-controller-769875db56-kpkmz" Oct 30 01:35:06.605862 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.605538 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04727d53-1a8f-4f08-9930-acfb77f05eb4-metrics-certs\") pod \"odh-model-controller-769875db56-kpkmz\" (UID: \"04727d53-1a8f-4f08-9930-acfb77f05eb4\") " pod="opendatahub/odh-model-controller-769875db56-kpkmz" Oct 30 01:35:06.607984 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.607960 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27395a7d-e510-484a-b2ec-9863321c7c4b-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-w7p7b\" (UID: \"27395a7d-e510-484a-b2ec-9863321c7c4b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b" Oct 30 01:35:06.608390 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.608369 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712fc2f5-d537-4e96-9801-963522329de7-tls-certs\") pod \"model-serving-api-75bd89fdfc-9fjjh\" (UID: \"712fc2f5-d537-4e96-9801-963522329de7\") " pod="opendatahub/model-serving-api-75bd89fdfc-9fjjh" Oct 30 01:35:06.628450 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.628414 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmr2r\" (UniqueName: \"kubernetes.io/projected/712fc2f5-d537-4e96-9801-963522329de7-kube-api-access-kmr2r\") pod \"model-serving-api-75bd89fdfc-9fjjh\" (UID: \"712fc2f5-d537-4e96-9801-963522329de7\") " pod="opendatahub/model-serving-api-75bd89fdfc-9fjjh" Oct 30 01:35:06.628450 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.628442 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v96t6\" (UniqueName: \"kubernetes.io/projected/27395a7d-e510-484a-b2ec-9863321c7c4b-kube-api-access-v96t6\") pod \"llmisvc-controller-manager-869d5dcc5d-w7p7b\" (UID: \"27395a7d-e510-484a-b2ec-9863321c7c4b\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b" Oct 30 01:35:06.649974 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.649941 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b" Oct 30 01:35:06.694760 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.694596 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-9fjjh" Oct 30 01:35:06.706537 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.706493 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04727d53-1a8f-4f08-9930-acfb77f05eb4-cert\") pod \"odh-model-controller-769875db56-kpkmz\" (UID: \"04727d53-1a8f-4f08-9930-acfb77f05eb4\") " pod="opendatahub/odh-model-controller-769875db56-kpkmz" Oct 30 01:35:06.706693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.706592 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzbvw\" (UniqueName: \"kubernetes.io/projected/04727d53-1a8f-4f08-9930-acfb77f05eb4-kube-api-access-qzbvw\") pod \"odh-model-controller-769875db56-kpkmz\" (UID: \"04727d53-1a8f-4f08-9930-acfb77f05eb4\") " pod="opendatahub/odh-model-controller-769875db56-kpkmz" Oct 30 01:35:06.706693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.706634 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04727d53-1a8f-4f08-9930-acfb77f05eb4-metrics-certs\") pod \"odh-model-controller-769875db56-kpkmz\" (UID: \"04727d53-1a8f-4f08-9930-acfb77f05eb4\") " pod="opendatahub/odh-model-controller-769875db56-kpkmz" Oct 30 01:35:06.711297 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.711264 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04727d53-1a8f-4f08-9930-acfb77f05eb4-metrics-certs\") pod \"odh-model-controller-769875db56-kpkmz\" (UID: \"04727d53-1a8f-4f08-9930-acfb77f05eb4\") " pod="opendatahub/odh-model-controller-769875db56-kpkmz" Oct 30 01:35:06.711615 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.711596 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04727d53-1a8f-4f08-9930-acfb77f05eb4-cert\") pod \"odh-model-controller-769875db56-kpkmz\" (UID: \"04727d53-1a8f-4f08-9930-acfb77f05eb4\") " pod="opendatahub/odh-model-controller-769875db56-kpkmz" Oct 30 01:35:06.730692 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.730657 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzbvw\" (UniqueName: \"kubernetes.io/projected/04727d53-1a8f-4f08-9930-acfb77f05eb4-kube-api-access-qzbvw\") pod \"odh-model-controller-769875db56-kpkmz\" (UID: \"04727d53-1a8f-4f08-9930-acfb77f05eb4\") " pod="opendatahub/odh-model-controller-769875db56-kpkmz" Oct 30 01:35:06.749640 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.749598 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-kpkmz" Oct 30 01:35:06.798504 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.798464 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:06.901735 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.901699 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-25r9k"] Oct 30 01:35:06.906519 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:35:06.906480 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7994cb01_5d42_4871_ae17_d9622123f971.slice/crio-a42bab34216607569750121011c0103c2986c8e851ff59c6b8a06ae86f1ddd84 WatchSource:0}: Error finding container a42bab34216607569750121011c0103c2986c8e851ff59c6b8a06ae86f1ddd84: Status 404 returned error can't find the container with id a42bab34216607569750121011c0103c2986c8e851ff59c6b8a06ae86f1ddd84 Oct 30 01:35:06.957254 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:06.957199 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b"] Oct 30 01:35:06.961511 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:35:06.961479 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod27395a7d_e510_484a_b2ec_9863321c7c4b.slice/crio-11bbca6be1849b7a5bca8fdfcbdf69c035a7e175d753adddeba2f66698473802 WatchSource:0}: Error finding container 11bbca6be1849b7a5bca8fdfcbdf69c035a7e175d753adddeba2f66698473802: Status 404 returned error can't find the container with id 11bbca6be1849b7a5bca8fdfcbdf69c035a7e175d753adddeba2f66698473802 Oct 30 01:35:07.008044 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:07.007946 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-9fjjh"] Oct 30 01:35:07.013133 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:35:07.013102 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712fc2f5_d537_4e96_9801_963522329de7.slice/crio-c89b2af884f220c87d37a175e90dc44463c9bc0402edb8d1e675ff132964960b WatchSource:0}: Error finding container c89b2af884f220c87d37a175e90dc44463c9bc0402edb8d1e675ff132964960b: Status 404 returned error can't find the container with id c89b2af884f220c87d37a175e90dc44463c9bc0402edb8d1e675ff132964960b Oct 30 01:35:07.056413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:07.056376 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-kpkmz"] Oct 30 01:35:07.061137 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:35:07.061110 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04727d53_1a8f_4f08_9930_acfb77f05eb4.slice/crio-db310aabe030d1bc7fbf897e1b909318772337a00f4b04a07847a414d4049ea1 WatchSource:0}: Error finding container db310aabe030d1bc7fbf897e1b909318772337a00f4b04a07847a414d4049ea1: Status 404 returned error can't find the container with id db310aabe030d1bc7fbf897e1b909318772337a00f4b04a07847a414d4049ea1 Oct 30 01:35:07.211902 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:07.211868 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-kpkmz" event={"ID":"04727d53-1a8f-4f08-9930-acfb77f05eb4","Type":"ContainerStarted","Data":"db310aabe030d1bc7fbf897e1b909318772337a00f4b04a07847a414d4049ea1"} Oct 30 01:35:07.213042 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:07.213015 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-25r9k" event={"ID":"7994cb01-5d42-4871-ae17-d9622123f971","Type":"ContainerStarted","Data":"a42bab34216607569750121011c0103c2986c8e851ff59c6b8a06ae86f1ddd84"} Oct 30 01:35:07.214142 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:07.214115 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-9fjjh" event={"ID":"712fc2f5-d537-4e96-9801-963522329de7","Type":"ContainerStarted","Data":"c89b2af884f220c87d37a175e90dc44463c9bc0402edb8d1e675ff132964960b"} Oct 30 01:35:07.215513 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:07.215480 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b" event={"ID":"27395a7d-e510-484a-b2ec-9863321c7c4b","Type":"ContainerStarted","Data":"11bbca6be1849b7a5bca8fdfcbdf69c035a7e175d753adddeba2f66698473802"} Oct 30 01:35:08.733970 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:08.733913 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:08.841589 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:08.841553 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:08.945460 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:08.945417 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:09.052920 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:09.052795 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:09.157333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:09.157297 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:10.249992 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:10.249911 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-9fjjh" event={"ID":"712fc2f5-d537-4e96-9801-963522329de7","Type":"ContainerStarted","Data":"c57441a5d3e6bbd80bd6187f7639604275cffb09e80bd4beb72fa4bb6f1a86c9"} Oct 30 01:35:10.249992 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:10.249991 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-9fjjh" Oct 30 01:35:10.251769 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:10.251739 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b" event={"ID":"27395a7d-e510-484a-b2ec-9863321c7c4b","Type":"ContainerStarted","Data":"e9e51d65a3e86e874b6c533adddd8194b30a28c34db69336c9e1f7f5bd71f69b"} Oct 30 01:35:10.251918 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:10.251830 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b" Oct 30 01:35:10.253535 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:10.253510 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-25r9k" event={"ID":"7994cb01-5d42-4871-ae17-d9622123f971","Type":"ContainerStarted","Data":"dc3d7761f5ecf4a6079a1109f28091fe9453d154e6f668bda763988b942f18f5"} Oct 30 01:35:10.253665 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:10.253649 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-25r9k" Oct 30 01:35:10.291778 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:10.291720 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-9fjjh" podStartSLOduration=1.6356761720000002 podStartE2EDuration="4.291702187s" totalImagesPullingTime="2.656026015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:10.290198624 +0000 UTC m=+339.868989705" watchObservedRunningTime="2026-10-30 01:35:10.291702187 +0000 UTC m=+339.870493283" Oct 30 01:35:10.330915 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:10.330857 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b" podStartSLOduration=1.482477659 podStartE2EDuration="4.330839445s" totalImagesPullingTime="2.848361786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:10.329751825 +0000 UTC m=+339.908542907" watchObservedRunningTime="2026-10-30 01:35:10.330839445 +0000 UTC m=+339.909630526" Oct 30 01:35:10.369845 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:10.369783 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-25r9k" podStartSLOduration=1.3697503 podStartE2EDuration="4.369767896s" totalImagesPullingTime="3.000017596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:10.369276632 +0000 UTC m=+339.948067714" watchObservedRunningTime="2026-10-30 01:35:10.369767896 +0000 UTC m=+339.948558977" Oct 30 01:35:11.260113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:11.260048 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-kpkmz" event={"ID":"04727d53-1a8f-4f08-9930-acfb77f05eb4","Type":"ContainerStarted","Data":"ce5135c4609a714fafe3c5486ed1df333c56a08bf7e5da216e5f0bae825ef209"} Oct 30 01:35:11.260598 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:11.260307 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-kpkmz" Oct 30 01:35:11.302330 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:11.302275 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-kpkmz" podStartSLOduration=1.835912271 podStartE2EDuration="5.302261102s" totalImagesPullingTime="3.466348831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:11.301271514 +0000 UTC m=+340.880062595" watchObservedRunningTime="2026-10-30 01:35:11.302261102 +0000 UTC m=+340.881052182" Oct 30 01:35:21.264553 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:21.264520 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-9fjjh" Oct 30 01:35:22.268029 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:22.267991 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-kpkmz" Oct 30 01:35:25.214949 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:35:25.214910 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:35:32.049251 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:32.049220 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:35:41.262788 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:41.262755 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b" Oct 30 01:35:41.266430 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:35:41.266405 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-25r9k" Oct 30 01:35:55.215191 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:35:55.215085 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:36:25.215953 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:36:25.215910 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:36:32.173384 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:36:32.173344 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:36:55.215722 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:36:55.215682 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:37:03.879940 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:37:03.879890 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:25.215272 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:37:25.215232 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:37:32.291392 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:37:32.291352 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:37:55.216153 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:37:55.216110 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:38:14.886904 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:38:14.886868 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:25.215973 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:38:25.215934 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:38:32.415536 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:38:32.415498 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:38:55.215504 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:38:55.215419 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:39:13.641372 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:39:13.641332 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:13.751387 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:39:13.751352 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:25.215805 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:39:25.215763 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:39:32.558047 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:39:32.558011 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:48.785806 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:39:48.785770 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:48.889932 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:39:48.889899 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:39:55.219497 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:39:55.219444 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:40:19.424372 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:40:19.424293 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:22.889343 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:40:22.889307 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:25.215460 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:40:25.215422 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:40:32.660299 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:40:32.660265 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:40:55.215814 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:40:55.215774 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:41:25.215734 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:41:25.215695 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:41:32.788334 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:41:32.788299 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:41:55.216346 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:41:55.216260 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:42:03.573356 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:42:03.573323 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:03.684743 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:42:03.684712 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:25.216232 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:42:25.216199 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:42:32.948078 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:42:32.948044 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:42:55.221697 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:42:55.221654 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:43:25.215379 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:43:25.215343 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:43:33.040640 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:43:33.040604 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:43:55.215890 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:43:55.215844 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:44:17.263034 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:17.263002 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:20.709600 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:20.709571 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-w5pkt_994b558f-0da2-400b-af37-ac1a0a691fdc/manager/0.log" Oct 30 01:44:20.945247 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:20.945211 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt"] Oct 30 01:44:20.945497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:20.945451 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt" podUID="994b558f-0da2-400b-af37-ac1a0a691fdc" containerName="manager" containerID="cri-o://cca2d07dbcdfa0da509f6acde4a9c8b139102732b6eaf9480b710a77dc301d01" gracePeriod=10 Oct 30 01:44:20.973718 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:20.973692 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l"] Oct 30 01:44:20.976936 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:20.976918 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l" Oct 30 01:44:21.010377 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.010349 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l"] Oct 30 01:44:21.029164 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.029128 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b4e4651-47b0-4440-b8a1-87d7fdb96f46-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rjj9l\" (UID: \"9b4e4651-47b0-4440-b8a1-87d7fdb96f46\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l" Oct 30 01:44:21.029291 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.029272 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srp6k\" (UniqueName: \"kubernetes.io/projected/9b4e4651-47b0-4440-b8a1-87d7fdb96f46-kube-api-access-srp6k\") pod \"kuadrant-operator-controller-manager-678d87766f-rjj9l\" (UID: \"9b4e4651-47b0-4440-b8a1-87d7fdb96f46\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l" Oct 30 01:44:21.130790 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.130752 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b4e4651-47b0-4440-b8a1-87d7fdb96f46-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rjj9l\" (UID: \"9b4e4651-47b0-4440-b8a1-87d7fdb96f46\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l" Oct 30 01:44:21.130923 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.130883 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srp6k\" (UniqueName: \"kubernetes.io/projected/9b4e4651-47b0-4440-b8a1-87d7fdb96f46-kube-api-access-srp6k\") pod \"kuadrant-operator-controller-manager-678d87766f-rjj9l\" (UID: \"9b4e4651-47b0-4440-b8a1-87d7fdb96f46\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l" Oct 30 01:44:21.131260 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.131233 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b4e4651-47b0-4440-b8a1-87d7fdb96f46-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rjj9l\" (UID: \"9b4e4651-47b0-4440-b8a1-87d7fdb96f46\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l" Oct 30 01:44:21.154176 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.154149 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srp6k\" (UniqueName: \"kubernetes.io/projected/9b4e4651-47b0-4440-b8a1-87d7fdb96f46-kube-api-access-srp6k\") pod \"kuadrant-operator-controller-manager-678d87766f-rjj9l\" (UID: \"9b4e4651-47b0-4440-b8a1-87d7fdb96f46\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l" Oct 30 01:44:21.188751 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.188724 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt" Oct 30 01:44:21.332054 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.331968 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994b558f-0da2-400b-af37-ac1a0a691fdc-kube-api-access-72jkp\" (UniqueName: \"kubernetes.io/projected/994b558f-0da2-400b-af37-ac1a0a691fdc-kube-api-access-72jkp\") pod \"994b558f-0da2-400b-af37-ac1a0a691fdc\" (UID: \"994b558f-0da2-400b-af37-ac1a0a691fdc\") " Oct 30 01:44:21.332054 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.332008 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/994b558f-0da2-400b-af37-ac1a0a691fdc-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/994b558f-0da2-400b-af37-ac1a0a691fdc-extensions-socket-volume\") pod \"994b558f-0da2-400b-af37-ac1a0a691fdc\" (UID: \"994b558f-0da2-400b-af37-ac1a0a691fdc\") " Oct 30 01:44:21.332427 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.332402 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/994b558f-0da2-400b-af37-ac1a0a691fdc-extensions-socket-volume" pod "994b558f-0da2-400b-af37-ac1a0a691fdc" (UID: "994b558f-0da2-400b-af37-ac1a0a691fdc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:21.334148 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.334124 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994b558f-0da2-400b-af37-ac1a0a691fdc-kube-api-access-72jkp" pod "994b558f-0da2-400b-af37-ac1a0a691fdc" (UID: "994b558f-0da2-400b-af37-ac1a0a691fdc"). InnerVolumeSpecName "kube-api-access-72jkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:21.341286 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.341266 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l" Oct 30 01:44:21.433000 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.432969 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72jkp\" (UniqueName: \"kubernetes.io/projected/994b558f-0da2-400b-af37-ac1a0a691fdc-kube-api-access-72jkp\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:44:21.433000 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.432997 5998 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/994b558f-0da2-400b-af37-ac1a0a691fdc-extensions-socket-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:44:21.659626 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.659540 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l"] Oct 30 01:44:21.665606 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:44:21.665579 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4e4651_47b0_4440_b8a1_87d7fdb96f46.slice/crio-ee1ce8f8e19c127af652747fe997446d9cccaf4ba369f9207a446a4eae3efc04 WatchSource:0}: Error finding container ee1ce8f8e19c127af652747fe997446d9cccaf4ba369f9207a446a4eae3efc04: Status 404 returned error can't find the container with id ee1ce8f8e19c127af652747fe997446d9cccaf4ba369f9207a446a4eae3efc04 Oct 30 01:44:21.911719 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.911674 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l" event={"ID":"9b4e4651-47b0-4440-b8a1-87d7fdb96f46","Type":"ContainerStarted","Data":"4f3f8d56a6896e21ec740dd58187a225dc5600a6b8043a40cc3fbd16275a7f66"} Oct 30 01:44:21.912202 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.911751 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l" Oct 30 01:44:21.912202 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.911776 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l" event={"ID":"9b4e4651-47b0-4440-b8a1-87d7fdb96f46","Type":"ContainerStarted","Data":"ee1ce8f8e19c127af652747fe997446d9cccaf4ba369f9207a446a4eae3efc04"} Oct 30 01:44:21.913080 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.913050 5998 generic.go:356] "Generic (PLEG): container finished" podID="994b558f-0da2-400b-af37-ac1a0a691fdc" containerID="cca2d07dbcdfa0da509f6acde4a9c8b139102732b6eaf9480b710a77dc301d01" exitCode=0 Oct 30 01:44:21.913206 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.913113 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt" Oct 30 01:44:21.913206 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.913118 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt" event={"ID":"994b558f-0da2-400b-af37-ac1a0a691fdc","Type":"ContainerDied","Data":"cca2d07dbcdfa0da509f6acde4a9c8b139102732b6eaf9480b710a77dc301d01"} Oct 30 01:44:21.913206 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.913166 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt" event={"ID":"994b558f-0da2-400b-af37-ac1a0a691fdc","Type":"ContainerDied","Data":"1f53f3ee939455b1413e8d515a99563e05477f5acbf63e637cb57ea6adf40211"} Oct 30 01:44:21.913206 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.913195 5998 scope.go:122] "RemoveContainer" containerID="cca2d07dbcdfa0da509f6acde4a9c8b139102732b6eaf9480b710a77dc301d01" Oct 30 01:44:21.922376 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.922359 5998 scope.go:122] "RemoveContainer" containerID="cca2d07dbcdfa0da509f6acde4a9c8b139102732b6eaf9480b710a77dc301d01" Oct 30 01:44:21.922662 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:44:21.922642 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cca2d07dbcdfa0da509f6acde4a9c8b139102732b6eaf9480b710a77dc301d01\": container with ID starting with cca2d07dbcdfa0da509f6acde4a9c8b139102732b6eaf9480b710a77dc301d01 not found: ID does not exist" containerID="cca2d07dbcdfa0da509f6acde4a9c8b139102732b6eaf9480b710a77dc301d01" Oct 30 01:44:21.922713 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.922674 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cca2d07dbcdfa0da509f6acde4a9c8b139102732b6eaf9480b710a77dc301d01"} err="failed to get container status \"cca2d07dbcdfa0da509f6acde4a9c8b139102732b6eaf9480b710a77dc301d01\": rpc error: code = NotFound desc = could not find container \"cca2d07dbcdfa0da509f6acde4a9c8b139102732b6eaf9480b710a77dc301d01\": container with ID starting with cca2d07dbcdfa0da509f6acde4a9c8b139102732b6eaf9480b710a77dc301d01 not found: ID does not exist" Oct 30 01:44:21.958309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:21.958262 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l" podStartSLOduration=1.958246623 podStartE2EDuration="1.958246623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:21.958061702 +0000 UTC m=+891.536852794" watchObservedRunningTime="2026-10-30 01:44:21.958246623 +0000 UTC m=+891.537037705" Oct 30 01:44:22.000223 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:22.000190 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt"] Oct 30 01:44:22.003738 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:22.003713 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-w5pkt"] Oct 30 01:44:22.536877 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:22.536843 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="994b558f-0da2-400b-af37-ac1a0a691fdc" path="/var/lib/kubelet/pods/994b558f-0da2-400b-af37-ac1a0a691fdc/volumes" Oct 30 01:44:25.216537 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:44:25.216502 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:44:32.921124 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:32.921070 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l" Oct 30 01:44:33.185965 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:33.185883 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:37.982137 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:37.982103 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:38.104438 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:38.104406 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:38.903502 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:38.903468 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:39.029231 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.029197 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:39.340324 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.340283 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:44:39.340778 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.340750 5998 cpu_manager.go:414] "containerMap: removing container" podUID="994b558f-0da2-400b-af37-ac1a0a691fdc" containerName="manager" Oct 30 01:44:39.340778 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.340775 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994b558f-0da2-400b-af37-ac1a0a691fdc" containerName="manager" Oct 30 01:44:39.340983 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.340881 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="994b558f-0da2-400b-af37-ac1a0a691fdc" containerName="manager" Oct 30 01:44:39.345631 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.345612 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-p2m8w" Oct 30 01:44:39.365740 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.365719 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 01:44:39.367216 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.367196 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:44:39.374615 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.374593 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-w4cxk\"" Oct 30 01:44:39.424228 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.424155 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:44:39.484451 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.484412 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/db342fa8-f998-43b9-8e0a-5da7612cdee2-config-file\") pod \"limitador-limitador-7c9b87d574-p2m8w\" (UID: \"db342fa8-f998-43b9-8e0a-5da7612cdee2\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p2m8w" Oct 30 01:44:39.484623 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.484504 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d62q2\" (UniqueName: \"kubernetes.io/projected/db342fa8-f998-43b9-8e0a-5da7612cdee2-kube-api-access-d62q2\") pod \"limitador-limitador-7c9b87d574-p2m8w\" (UID: \"db342fa8-f998-43b9-8e0a-5da7612cdee2\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p2m8w" Oct 30 01:44:39.585501 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.585460 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/db342fa8-f998-43b9-8e0a-5da7612cdee2-config-file\") pod \"limitador-limitador-7c9b87d574-p2m8w\" (UID: \"db342fa8-f998-43b9-8e0a-5da7612cdee2\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p2m8w" Oct 30 01:44:39.585673 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.585518 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d62q2\" (UniqueName: \"kubernetes.io/projected/db342fa8-f998-43b9-8e0a-5da7612cdee2-kube-api-access-d62q2\") pod \"limitador-limitador-7c9b87d574-p2m8w\" (UID: \"db342fa8-f998-43b9-8e0a-5da7612cdee2\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p2m8w" Oct 30 01:44:39.586111 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.586065 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/db342fa8-f998-43b9-8e0a-5da7612cdee2-config-file\") pod \"limitador-limitador-7c9b87d574-p2m8w\" (UID: \"db342fa8-f998-43b9-8e0a-5da7612cdee2\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p2m8w" Oct 30 01:44:39.608069 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.608002 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d62q2\" (UniqueName: \"kubernetes.io/projected/db342fa8-f998-43b9-8e0a-5da7612cdee2-kube-api-access-d62q2\") pod \"limitador-limitador-7c9b87d574-p2m8w\" (UID: \"db342fa8-f998-43b9-8e0a-5da7612cdee2\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p2m8w" Oct 30 01:44:39.658905 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.658866 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-p2m8w" Oct 30 01:44:39.969377 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.969344 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:44:39.973346 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:44:39.973307 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb342fa8_f998_43b9_8e0a_5da7612cdee2.slice/crio-cef4289d86fc9e8fbb44af3a36d70816f3d486b185a79641563aad0e969bbc08 WatchSource:0}: Error finding container cef4289d86fc9e8fbb44af3a36d70816f3d486b185a79641563aad0e969bbc08: Status 404 returned error can't find the container with id cef4289d86fc9e8fbb44af3a36d70816f3d486b185a79641563aad0e969bbc08 Oct 30 01:44:39.975848 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:39.975831 5998 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:44:40.003100 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:40.003059 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-p2m8w" event={"ID":"db342fa8-f998-43b9-8e0a-5da7612cdee2","Type":"ContainerStarted","Data":"cef4289d86fc9e8fbb44af3a36d70816f3d486b185a79641563aad0e969bbc08"} Oct 30 01:44:41.010039 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:41.010008 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-p2m8w" event={"ID":"db342fa8-f998-43b9-8e0a-5da7612cdee2","Type":"ContainerStarted","Data":"16ff0e6dd417f23fc7c5c5d4d7b88b210198b07dfbe82882571af3cf039ea85c"} Oct 30 01:44:41.010443 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:41.010071 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-p2m8w" Oct 30 01:44:41.051571 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:41.051524 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-p2m8w" podStartSLOduration=1.160203432 podStartE2EDuration="2.051512242s" totalImagesPullingTime="891.30881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:41.050372029 +0000 UTC m=+910.629163111" watchObservedRunningTime="2026-10-30 01:44:41.051512242 +0000 UTC m=+910.630303322" Oct 30 01:44:52.015845 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:52.015771 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-p2m8w" Oct 30 01:44:54.712441 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:54.712407 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:54.857664 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:54.857631 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:44:55.215482 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:44:55.215442 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:44:55.523622 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:44:55.523543 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:14.204416 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:45:14.204375 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:14.323787 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:45:14.323730 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:14.429859 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:45:14.429826 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:14.542142 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:45:14.542039 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:17.888657 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:45:17.888620 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:18.000210 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:45:18.000170 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:19.851281 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:45:19.851245 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:19.957195 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:45:19.957163 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:25.216266 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:45:25.216226 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:45:33.280764 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:45:33.280733 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:49.069774 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:45:49.069738 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:49.202249 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:45:49.202210 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:50.533848 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:45:50.533807 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:50.647741 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:45:50.647706 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:50.755590 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:45:50.755557 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:50.862834 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:45:50.862759 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:45:52.479313 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:45:52.479279 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:45:55.215861 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:45:55.215821 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:46:01.111368 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:01.111332 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:01.329618 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:01.329579 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:04.593489 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:04.593450 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-wptrw"] Oct 30 01:46:04.597280 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:04.597259 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wptrw" Oct 30 01:46:04.618026 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:04.617996 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-wptrw"] Oct 30 01:46:04.715947 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:04.715900 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a3c5537-f83e-412e-a917-27fd46cfe258-cert\") pod \"llmisvc-controller-manager-6466c554d8-wptrw\" (UID: \"1a3c5537-f83e-412e-a917-27fd46cfe258\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-wptrw" Oct 30 01:46:04.716195 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:04.715954 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4bbr\" (UniqueName: \"kubernetes.io/projected/1a3c5537-f83e-412e-a917-27fd46cfe258-kube-api-access-m4bbr\") pod \"llmisvc-controller-manager-6466c554d8-wptrw\" (UID: \"1a3c5537-f83e-412e-a917-27fd46cfe258\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-wptrw" Oct 30 01:46:04.817273 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:04.817229 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a3c5537-f83e-412e-a917-27fd46cfe258-cert\") pod \"llmisvc-controller-manager-6466c554d8-wptrw\" (UID: \"1a3c5537-f83e-412e-a917-27fd46cfe258\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-wptrw" Oct 30 01:46:04.817463 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:04.817281 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4bbr\" (UniqueName: \"kubernetes.io/projected/1a3c5537-f83e-412e-a917-27fd46cfe258-kube-api-access-m4bbr\") pod \"llmisvc-controller-manager-6466c554d8-wptrw\" (UID: \"1a3c5537-f83e-412e-a917-27fd46cfe258\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-wptrw" Oct 30 01:46:04.819783 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:04.819760 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a3c5537-f83e-412e-a917-27fd46cfe258-cert\") pod \"llmisvc-controller-manager-6466c554d8-wptrw\" (UID: \"1a3c5537-f83e-412e-a917-27fd46cfe258\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-wptrw" Oct 30 01:46:04.840458 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:04.840420 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4bbr\" (UniqueName: \"kubernetes.io/projected/1a3c5537-f83e-412e-a917-27fd46cfe258-kube-api-access-m4bbr\") pod \"llmisvc-controller-manager-6466c554d8-wptrw\" (UID: \"1a3c5537-f83e-412e-a917-27fd46cfe258\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-wptrw" Oct 30 01:46:04.910314 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:04.910221 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wptrw" Oct 30 01:46:05.211765 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:05.211699 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-wptrw"] Oct 30 01:46:05.216334 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:46:05.216306 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3c5537_f83e_412e_a917_27fd46cfe258.slice/crio-1a025a836171db4dbb340661c464a4fcf6a224f69dffddfd4c5c7c6c1f87f87f WatchSource:0}: Error finding container 1a025a836171db4dbb340661c464a4fcf6a224f69dffddfd4c5c7c6c1f87f87f: Status 404 returned error can't find the container with id 1a025a836171db4dbb340661c464a4fcf6a224f69dffddfd4c5c7c6c1f87f87f Oct 30 01:46:05.415411 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:05.415373 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wptrw" event={"ID":"1a3c5537-f83e-412e-a917-27fd46cfe258","Type":"ContainerStarted","Data":"1a025a836171db4dbb340661c464a4fcf6a224f69dffddfd4c5c7c6c1f87f87f"} Oct 30 01:46:06.421871 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:06.421835 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wptrw" event={"ID":"1a3c5537-f83e-412e-a917-27fd46cfe258","Type":"ContainerStarted","Data":"e33f13d491fa9e9c5098da364a624387e09a80741f31b91fe664d163887d4794"} Oct 30 01:46:06.422280 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:06.421898 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wptrw" Oct 30 01:46:06.463246 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:06.463193 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wptrw" podStartSLOduration=1.775252359 podStartE2EDuration="2.463177065s" totalImagesPullingTime="687.924706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:06.462006488 +0000 UTC m=+996.040797569" watchObservedRunningTime="2026-10-30 01:46:06.463177065 +0000 UTC m=+996.041968145" Oct 30 01:46:25.215704 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:46:25.215662 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:46:33.406833 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:33.406801 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:37.428923 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:37.428890 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-wptrw" Oct 30 01:46:37.499528 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:37.499486 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b"] Oct 30 01:46:37.499758 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:37.499703 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b" podUID="27395a7d-e510-484a-b2ec-9863321c7c4b" containerName="manager" containerID="cri-o://e9e51d65a3e86e874b6c533adddd8194b30a28c34db69336c9e1f7f5bd71f69b" gracePeriod=30 Oct 30 01:46:37.750374 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:37.750347 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b" Oct 30 01:46:37.803604 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:37.803571 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27395a7d-e510-484a-b2ec-9863321c7c4b-kube-api-access-v96t6\" (UniqueName: \"kubernetes.io/projected/27395a7d-e510-484a-b2ec-9863321c7c4b-kube-api-access-v96t6\") pod \"27395a7d-e510-484a-b2ec-9863321c7c4b\" (UID: \"27395a7d-e510-484a-b2ec-9863321c7c4b\") " Oct 30 01:46:37.803759 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:37.803667 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27395a7d-e510-484a-b2ec-9863321c7c4b-cert\" (UniqueName: \"kubernetes.io/secret/27395a7d-e510-484a-b2ec-9863321c7c4b-cert\") pod \"27395a7d-e510-484a-b2ec-9863321c7c4b\" (UID: \"27395a7d-e510-484a-b2ec-9863321c7c4b\") " Oct 30 01:46:37.805776 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:37.805747 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27395a7d-e510-484a-b2ec-9863321c7c4b-cert" pod "27395a7d-e510-484a-b2ec-9863321c7c4b" (UID: "27395a7d-e510-484a-b2ec-9863321c7c4b"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:37.805877 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:37.805807 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27395a7d-e510-484a-b2ec-9863321c7c4b-kube-api-access-v96t6" pod "27395a7d-e510-484a-b2ec-9863321c7c4b" (UID: "27395a7d-e510-484a-b2ec-9863321c7c4b"). InnerVolumeSpecName "kube-api-access-v96t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:37.904404 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:37.904375 5998 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27395a7d-e510-484a-b2ec-9863321c7c4b-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:46:37.904404 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:37.904401 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v96t6\" (UniqueName: \"kubernetes.io/projected/27395a7d-e510-484a-b2ec-9863321c7c4b-kube-api-access-v96t6\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:46:38.579344 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:38.579311 5998 generic.go:356] "Generic (PLEG): container finished" podID="27395a7d-e510-484a-b2ec-9863321c7c4b" containerID="e9e51d65a3e86e874b6c533adddd8194b30a28c34db69336c9e1f7f5bd71f69b" exitCode=0 Oct 30 01:46:38.579718 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:38.579350 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b" event={"ID":"27395a7d-e510-484a-b2ec-9863321c7c4b","Type":"ContainerDied","Data":"e9e51d65a3e86e874b6c533adddd8194b30a28c34db69336c9e1f7f5bd71f69b"} Oct 30 01:46:38.579718 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:38.579365 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b" Oct 30 01:46:38.579718 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:38.579403 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b" event={"ID":"27395a7d-e510-484a-b2ec-9863321c7c4b","Type":"ContainerDied","Data":"11bbca6be1849b7a5bca8fdfcbdf69c035a7e175d753adddeba2f66698473802"} Oct 30 01:46:38.579718 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:38.579426 5998 scope.go:122] "RemoveContainer" containerID="e9e51d65a3e86e874b6c533adddd8194b30a28c34db69336c9e1f7f5bd71f69b" Oct 30 01:46:38.588052 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:38.588035 5998 scope.go:122] "RemoveContainer" containerID="e9e51d65a3e86e874b6c533adddd8194b30a28c34db69336c9e1f7f5bd71f69b" Oct 30 01:46:38.588366 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:46:38.588347 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e51d65a3e86e874b6c533adddd8194b30a28c34db69336c9e1f7f5bd71f69b\": container with ID starting with e9e51d65a3e86e874b6c533adddd8194b30a28c34db69336c9e1f7f5bd71f69b not found: ID does not exist" containerID="e9e51d65a3e86e874b6c533adddd8194b30a28c34db69336c9e1f7f5bd71f69b" Oct 30 01:46:38.588430 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:38.588380 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e51d65a3e86e874b6c533adddd8194b30a28c34db69336c9e1f7f5bd71f69b"} err="failed to get container status \"e9e51d65a3e86e874b6c533adddd8194b30a28c34db69336c9e1f7f5bd71f69b\": rpc error: code = NotFound desc = could not find container \"e9e51d65a3e86e874b6c533adddd8194b30a28c34db69336c9e1f7f5bd71f69b\": container with ID starting with e9e51d65a3e86e874b6c533adddd8194b30a28c34db69336c9e1f7f5bd71f69b not found: ID does not exist" Oct 30 01:46:38.635477 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:38.635445 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b"] Oct 30 01:46:38.646917 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:38.646894 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w7p7b"] Oct 30 01:46:38.925766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:38.925734 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:39.030814 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:39.030784 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:40.536983 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:40.536948 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27395a7d-e510-484a-b2ec-9863321c7c4b" path="/var/lib/kubelet/pods/27395a7d-e510-484a-b2ec-9863321c7c4b/volumes" Oct 30 01:46:43.167716 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:43.167680 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:43.282145 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:46:43.282113 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:46:55.215977 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:46:55.215925 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:47:07.618528 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:07.618481 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:08.100363 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:08.100327 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:09.315558 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:09.315510 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:09.429115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:09.429044 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:13.250657 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:13.250623 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:14.253054 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:14.253016 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:47:15.753186 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:15.753150 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:17.148154 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:17.148112 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:17.880191 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:17.880156 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:47:18.650576 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:18.650540 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:21.166387 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:21.166351 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:22.598910 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:22.598874 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:47:22.762712 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:22.762672 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:24.052468 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:24.052434 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:25.217206 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:47:25.217170 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:47:25.418814 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:25.418775 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:29.000083 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:29.000040 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:47:33.531074 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:33.531038 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:40.200696 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.200659 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v"] Oct 30 01:47:40.201222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.201199 5998 cpu_manager.go:414] "containerMap: removing container" podUID="27395a7d-e510-484a-b2ec-9863321c7c4b" containerName="manager" Oct 30 01:47:40.201287 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.201225 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27395a7d-e510-484a-b2ec-9863321c7c4b" containerName="manager" Oct 30 01:47:40.201377 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.201360 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="27395a7d-e510-484a-b2ec-9863321c7c4b" containerName="manager" Oct 30 01:47:40.206067 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.206046 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.235690 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.235657 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v"] Oct 30 01:47:40.249970 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.249938 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.250162 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.249981 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.250162 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.250005 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.250263 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.250168 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqfr8\" (UniqueName: \"kubernetes.io/projected/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-kube-api-access-tqfr8\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.250263 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.250189 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.250339 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.250312 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.250400 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.250378 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.250451 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.250442 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.250556 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.250536 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.301654 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.301621 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5dmpm\"" Oct 30 01:47:40.351804 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.351767 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.352015 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.351840 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.352015 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.351860 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.352015 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.351880 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.352015 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.351915 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqfr8\" (UniqueName: \"kubernetes.io/projected/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-kube-api-access-tqfr8\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.352015 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.351945 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.352015 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.351986 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.352468 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.352030 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.352468 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.352081 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.352468 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.352433 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.352468 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.352453 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.352687 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.352547 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.352805 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.352782 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.352914 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.352893 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.354503 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.354475 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.354649 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.354630 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.374978 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.374946 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.379929 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.379673 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqfr8\" (UniqueName: \"kubernetes.io/projected/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-kube-api-access-tqfr8\") pod \"maas-default-gateway-openshift-default-75d7c785f-57s4v\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.519406 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.519323 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:40.839150 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.839061 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v"] Oct 30 01:47:40.844846 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:47:40.844814 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaaec13d_e62d_4450_864c_d9e3fb7ca8d4.slice/crio-968fdf21f5532bd038957d8026bf32548f402eda004493265226863cfada2d7d WatchSource:0}: Error finding container 968fdf21f5532bd038957d8026bf32548f402eda004493265226863cfada2d7d: Status 404 returned error can't find the container with id 968fdf21f5532bd038957d8026bf32548f402eda004493265226863cfada2d7d Oct 30 01:47:40.846861 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.846828 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:47:40.846918 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.846895 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:47:40.846956 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.846926 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:47:40.888497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:40.888470 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" event={"ID":"caaec13d-e62d-4450-864c-d9e3fb7ca8d4","Type":"ContainerStarted","Data":"968fdf21f5532bd038957d8026bf32548f402eda004493265226863cfada2d7d"} Oct 30 01:47:41.895494 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:41.895408 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" event={"ID":"caaec13d-e62d-4450-864c-d9e3fb7ca8d4","Type":"ContainerStarted","Data":"dd5450b72b56f3408b6dd3dd3799b0306b2253d06e54c4d3421bd5ecd1bcfd4e"} Oct 30 01:47:41.953604 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:41.953546 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" podStartSLOduration=1.953530179 podStartE2EDuration="1.953530179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:41.953169158 +0000 UTC m=+1091.531960240" watchObservedRunningTime="2026-10-30 01:47:41.953530179 +0000 UTC m=+1091.532321259" Oct 30 01:47:42.523114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:42.522458 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:42.526120 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:42.525326 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-57s4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 01:47:42.526120 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:42.525407 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" podUID="caaec13d-e62d-4450-864c-d9e3fb7ca8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 01:47:43.073896 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:43.073861 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:43.189827 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:43.189791 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:43.524115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:43.523498 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-57s4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 01:47:43.524115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:43.523566 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" podUID="caaec13d-e62d-4450-864c-d9e3fb7ca8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 01:47:44.521534 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:44.521496 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-57s4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 01:47:44.522305 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:44.522269 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" podUID="caaec13d-e62d-4450-864c-d9e3fb7ca8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 01:47:44.597352 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:44.597317 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:44.728245 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:44.728210 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:47:45.521587 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:45.521550 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-57s4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 01:47:45.522242 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:45.522211 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" podUID="caaec13d-e62d-4450-864c-d9e3fb7ca8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 01:47:46.523567 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:46.523521 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-57s4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 01:47:46.524195 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:46.524160 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" podUID="caaec13d-e62d-4450-864c-d9e3fb7ca8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 01:47:47.519761 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:47.519721 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-57s4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 01:47:47.519969 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:47.519780 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" podUID="caaec13d-e62d-4450-864c-d9e3fb7ca8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 01:47:49.135360 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:49.135317 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:49.135846 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:49.135698 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:49.136646 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:49.136624 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 01:47:55.216287 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:47:55.216235 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:47:55.243275 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.243244 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk"] Oct 30 01:47:55.246702 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.246681 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.274895 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.274842 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk"] Oct 30 01:47:55.422528 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.422488 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.422528 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.422528 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.422762 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.422554 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.422762 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.422669 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.422762 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.422706 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.422762 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.422732 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6x6g\" (UniqueName: \"kubernetes.io/projected/94053f0f-2fd8-4ab0-914a-256ddadb5d85-kube-api-access-m6x6g\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.422909 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.422804 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.422909 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.422896 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.422978 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.422935 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.523942 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.523841 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.523942 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.523902 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.523942 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.523943 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.524276 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.523965 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.524276 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.523985 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.524276 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.524023 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.524276 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.524051 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.524276 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.524081 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6x6g\" (UniqueName: \"kubernetes.io/projected/94053f0f-2fd8-4ab0-914a-256ddadb5d85-kube-api-access-m6x6g\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.524276 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.524141 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.524544 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.524467 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.524544 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.524489 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.524744 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.524715 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.524882 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.524766 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.524882 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.524864 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.526284 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.526262 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.526516 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.526495 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.547263 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.547236 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.547396 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.547318 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6x6g\" (UniqueName: \"kubernetes.io/projected/94053f0f-2fd8-4ab0-914a-256ddadb5d85-kube-api-access-m6x6g\") pod \"maas-default-gateway-openshift-default-75d7c785f-zghmk\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.561133 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.561112 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:55.881173 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.881070 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk"] Oct 30 01:47:55.885526 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:47:55.885498 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94053f0f_2fd8_4ab0_914a_256ddadb5d85.slice/crio-3a6c94ab901a1f5f028b5dcd2358dc1fa1fd39ad31bdde2ad89f3fe51ee23676 WatchSource:0}: Error finding container 3a6c94ab901a1f5f028b5dcd2358dc1fa1fd39ad31bdde2ad89f3fe51ee23676: Status 404 returned error can't find the container with id 3a6c94ab901a1f5f028b5dcd2358dc1fa1fd39ad31bdde2ad89f3fe51ee23676 Oct 30 01:47:55.887792 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.887762 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:47:55.887861 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.887826 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:47:55.887861 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.887855 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:47:55.994681 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.994641 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" event={"ID":"94053f0f-2fd8-4ab0-914a-256ddadb5d85","Type":"ContainerStarted","Data":"3027f63794dc2178391b57d4f2c91a0592db3cf295ea9b87658997a8e772778b"} Oct 30 01:47:55.995064 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:55.994713 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" event={"ID":"94053f0f-2fd8-4ab0-914a-256ddadb5d85","Type":"ContainerStarted","Data":"3a6c94ab901a1f5f028b5dcd2358dc1fa1fd39ad31bdde2ad89f3fe51ee23676"} Oct 30 01:47:56.038474 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:56.038414 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" podStartSLOduration=1.038398378 podStartE2EDuration="1.038398378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:56.037966918 +0000 UTC m=+1105.616758002" watchObservedRunningTime="2026-10-30 01:47:56.038398378 +0000 UTC m=+1105.617189518" Oct 30 01:47:56.561420 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:56.561385 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:47:56.563066 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:56.563041 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zghmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 01:47:56.563217 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:56.563085 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" podUID="94053f0f-2fd8-4ab0-914a-256ddadb5d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 01:47:57.562279 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:57.562236 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zghmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 01:47:57.562777 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:57.562296 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" podUID="94053f0f-2fd8-4ab0-914a-256ddadb5d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 01:47:58.561919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:58.561890 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zghmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 01:47:58.562113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:58.561943 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" podUID="94053f0f-2fd8-4ab0-914a-256ddadb5d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 01:47:59.562390 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:59.562343 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zghmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 01:47:59.562858 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:47:59.562407 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" podUID="94053f0f-2fd8-4ab0-914a-256ddadb5d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 01:48:00.562244 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:00.562210 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zghmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 01:48:00.562420 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:00.562271 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" podUID="94053f0f-2fd8-4ab0-914a-256ddadb5d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 01:48:01.561794 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:01.561755 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zghmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 01:48:01.561983 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:01.561815 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" podUID="94053f0f-2fd8-4ab0-914a-256ddadb5d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 01:48:02.562378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:02.562335 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zghmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 01:48:02.562805 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:02.562404 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" podUID="94053f0f-2fd8-4ab0-914a-256ddadb5d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 01:48:04.125361 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:04.125320 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:48:04.125847 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:04.125689 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:48:04.126680 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:04.126656 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 01:48:05.389658 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:05.389621 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:14.707468 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:14.707430 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:14.855737 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:14.855697 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:15.070553 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:15.070475 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:15.186295 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:15.186262 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:15.363760 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:15.363670 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:15.491169 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:15.491123 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:15.599832 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:15.599795 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:15.755884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:15.755851 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:15.865304 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:15.865268 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:16.000624 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:16.000584 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:16.622420 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:16.622383 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:17.013066 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:17.013036 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:17.377533 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:17.377444 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:17.913261 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:17.913223 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:18.042447 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:18.042407 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:18.182894 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:18.182813 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:18.398157 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:18.398120 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:19.620269 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:19.620221 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:19.729573 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:19.729524 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:20.191276 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:20.191243 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:20.296259 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:20.296227 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:20.950362 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:20.950319 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:21.070115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:21.070068 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:21.203657 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:21.203586 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:21.342023 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:21.341984 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:21.472073 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:21.471969 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:22.312694 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:22.312654 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:22.429609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:22.429578 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:23.115037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:23.115002 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:23.229284 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:23.229239 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:23.618786 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:23.618754 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:23.927442 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:23.927411 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:24.034235 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:24.034202 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:24.234835 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:24.234747 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:24.718234 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:24.718202 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:24.828715 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:24.828657 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:24.942299 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:24.942262 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:25.006737 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:25.006645 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:48:25.216686 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:48:25.216619 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:48:26.538299 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:26.538270 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:26.647029 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:26.646996 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:26.754708 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:26.754672 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:28.256514 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:28.256480 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:28.364684 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:28.364649 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:28.483294 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:28.483263 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:28.606731 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:28.606644 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:30.218236 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:30.218180 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:30.336683 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:30.336632 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:30.456660 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:30.456590 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:32.045984 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:32.045950 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:33.673195 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:33.673160 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:33.774712 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:33.774676 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:48:33.856005 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:33.855970 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:35.738602 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:35.738540 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:37.591038 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:37.590992 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:37.710746 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:37.710714 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:37.876235 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:37.876081 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:39.060912 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.060879 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf"] Oct 30 01:48:39.065484 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.065448 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf" Oct 30 01:48:39.079737 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.079713 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:48:39.091415 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.091386 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:48:39.091415 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.091391 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:48:39.091629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.091455 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:48:39.093331 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.093308 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf"] Oct 30 01:48:39.231436 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.231385 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9z6c\" (UniqueName: \"kubernetes.io/projected/9e55b0a1-21ae-406d-af93-b1846b5e891f-kube-api-access-x9z6c\") pod \"maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf\" (UID: \"9e55b0a1-21ae-406d-af93-b1846b5e891f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf" Oct 30 01:48:39.231637 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.231559 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e55b0a1-21ae-406d-af93-b1846b5e891f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf\" (UID: \"9e55b0a1-21ae-406d-af93-b1846b5e891f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf" Oct 30 01:48:39.231637 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.231623 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e55b0a1-21ae-406d-af93-b1846b5e891f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf\" (UID: \"9e55b0a1-21ae-406d-af93-b1846b5e891f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf" Oct 30 01:48:39.333054 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.332967 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9z6c\" (UniqueName: \"kubernetes.io/projected/9e55b0a1-21ae-406d-af93-b1846b5e891f-kube-api-access-x9z6c\") pod \"maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf\" (UID: \"9e55b0a1-21ae-406d-af93-b1846b5e891f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf" Oct 30 01:48:39.333242 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.333101 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e55b0a1-21ae-406d-af93-b1846b5e891f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf\" (UID: \"9e55b0a1-21ae-406d-af93-b1846b5e891f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf" Oct 30 01:48:39.333242 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.333151 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e55b0a1-21ae-406d-af93-b1846b5e891f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf\" (UID: \"9e55b0a1-21ae-406d-af93-b1846b5e891f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf" Oct 30 01:48:39.333754 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.333728 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e55b0a1-21ae-406d-af93-b1846b5e891f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf\" (UID: \"9e55b0a1-21ae-406d-af93-b1846b5e891f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf" Oct 30 01:48:39.335463 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.335438 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e55b0a1-21ae-406d-af93-b1846b5e891f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf\" (UID: \"9e55b0a1-21ae-406d-af93-b1846b5e891f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf" Oct 30 01:48:39.363963 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.363929 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9z6c\" (UniqueName: \"kubernetes.io/projected/9e55b0a1-21ae-406d-af93-b1846b5e891f-kube-api-access-x9z6c\") pod \"maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf\" (UID: \"9e55b0a1-21ae-406d-af93-b1846b5e891f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf" Oct 30 01:48:39.380479 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.380454 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf" Oct 30 01:48:39.575279 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.575240 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:39.682802 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:39.682769 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf"] Oct 30 01:48:39.687825 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:48:39.687797 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e55b0a1_21ae_406d_af93_b1846b5e891f.slice/crio-1025d89048bb02fbd0eb015d25e6533b41ab5527fec9938382b56998465f82ce WatchSource:0}: Error finding container 1025d89048bb02fbd0eb015d25e6533b41ab5527fec9938382b56998465f82ce: Status 404 returned error can't find the container with id 1025d89048bb02fbd0eb015d25e6533b41ab5527fec9938382b56998465f82ce Oct 30 01:48:40.251491 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:40.251451 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf" event={"ID":"9e55b0a1-21ae-406d-af93-b1846b5e891f","Type":"ContainerStarted","Data":"1025d89048bb02fbd0eb015d25e6533b41ab5527fec9938382b56998465f82ce"} Oct 30 01:48:41.596499 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:41.596447 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:42.295028 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:42.294988 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:48:43.269412 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:43.269320 5998 generic.go:356] "Generic (PLEG): container finished" podID="9e55b0a1-21ae-406d-af93-b1846b5e891f" containerID="08dcfdb614c82903cf9eeb553e22ccb173de6399a2d65c7b6fe247bb92d1192d" exitCode=0 Oct 30 01:48:43.269412 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:43.269358 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf" event={"ID":"9e55b0a1-21ae-406d-af93-b1846b5e891f","Type":"ContainerDied","Data":"08dcfdb614c82903cf9eeb553e22ccb173de6399a2d65c7b6fe247bb92d1192d"} Oct 30 01:48:43.473435 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:43.473406 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:44.305375 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.305340 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92"] Oct 30 01:48:44.309082 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.309057 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92" Oct 30 01:48:44.331650 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.331618 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92"] Oct 30 01:48:44.380638 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.380607 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2ae791-7e6e-493c-8927-7116f567eb24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-61b6469a-5ds92\" (UID: \"9e2ae791-7e6e-493c-8927-7116f567eb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92" Oct 30 01:48:44.380810 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.380666 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2ae791-7e6e-493c-8927-7116f567eb24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-61b6469a-5ds92\" (UID: \"9e2ae791-7e6e-493c-8927-7116f567eb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92" Oct 30 01:48:44.380870 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.380814 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmps6\" (UniqueName: \"kubernetes.io/projected/9e2ae791-7e6e-493c-8927-7116f567eb24-kube-api-access-rmps6\") pod \"maas-api-revoke-keys-e2e-ait-61b6469a-5ds92\" (UID: \"9e2ae791-7e6e-493c-8927-7116f567eb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92" Oct 30 01:48:44.427570 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.427536 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf" Oct 30 01:48:44.477468 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.477433 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:44.481667 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.481641 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e55b0a1-21ae-406d-af93-b1846b5e891f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e55b0a1-21ae-406d-af93-b1846b5e891f-maas-api-openshift-ca\") pod \"9e55b0a1-21ae-406d-af93-b1846b5e891f\" (UID: \"9e55b0a1-21ae-406d-af93-b1846b5e891f\") " Oct 30 01:48:44.481839 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.481763 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e55b0a1-21ae-406d-af93-b1846b5e891f-kube-api-access-x9z6c\" (UniqueName: \"kubernetes.io/projected/9e55b0a1-21ae-406d-af93-b1846b5e891f-kube-api-access-x9z6c\") pod \"9e55b0a1-21ae-406d-af93-b1846b5e891f\" (UID: \"9e55b0a1-21ae-406d-af93-b1846b5e891f\") " Oct 30 01:48:44.481839 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.481812 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e55b0a1-21ae-406d-af93-b1846b5e891f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e55b0a1-21ae-406d-af93-b1846b5e891f-maas-api-xks-ca\") pod \"9e55b0a1-21ae-406d-af93-b1846b5e891f\" (UID: \"9e55b0a1-21ae-406d-af93-b1846b5e891f\") " Oct 30 01:48:44.481948 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.481937 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2ae791-7e6e-493c-8927-7116f567eb24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-61b6469a-5ds92\" (UID: \"9e2ae791-7e6e-493c-8927-7116f567eb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92" Oct 30 01:48:44.482051 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.482035 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmps6\" (UniqueName: \"kubernetes.io/projected/9e2ae791-7e6e-493c-8927-7116f567eb24-kube-api-access-rmps6\") pod \"maas-api-revoke-keys-e2e-ait-61b6469a-5ds92\" (UID: \"9e2ae791-7e6e-493c-8927-7116f567eb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92" Oct 30 01:48:44.482153 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.482055 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e55b0a1-21ae-406d-af93-b1846b5e891f-maas-api-openshift-ca" pod "9e55b0a1-21ae-406d-af93-b1846b5e891f" (UID: "9e55b0a1-21ae-406d-af93-b1846b5e891f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:48:44.482406 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.482367 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2ae791-7e6e-493c-8927-7116f567eb24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-61b6469a-5ds92\" (UID: \"9e2ae791-7e6e-493c-8927-7116f567eb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92" Oct 30 01:48:44.482502 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.482483 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e55b0a1-21ae-406d-af93-b1846b5e891f-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:48:44.483084 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.483040 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2ae791-7e6e-493c-8927-7116f567eb24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-61b6469a-5ds92\" (UID: \"9e2ae791-7e6e-493c-8927-7116f567eb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92" Oct 30 01:48:44.484283 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.484248 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e55b0a1-21ae-406d-af93-b1846b5e891f-maas-api-xks-ca" pod "9e55b0a1-21ae-406d-af93-b1846b5e891f" (UID: "9e55b0a1-21ae-406d-af93-b1846b5e891f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:48:44.484395 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.484371 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e55b0a1-21ae-406d-af93-b1846b5e891f-kube-api-access-x9z6c" pod "9e55b0a1-21ae-406d-af93-b1846b5e891f" (UID: "9e55b0a1-21ae-406d-af93-b1846b5e891f"). InnerVolumeSpecName "kube-api-access-x9z6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:48:44.485171 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.485147 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2ae791-7e6e-493c-8927-7116f567eb24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-61b6469a-5ds92\" (UID: \"9e2ae791-7e6e-493c-8927-7116f567eb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92" Oct 30 01:48:44.504736 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.504702 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmps6\" (UniqueName: \"kubernetes.io/projected/9e2ae791-7e6e-493c-8927-7116f567eb24-kube-api-access-rmps6\") pod \"maas-api-revoke-keys-e2e-ait-61b6469a-5ds92\" (UID: \"9e2ae791-7e6e-493c-8927-7116f567eb24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92" Oct 30 01:48:44.583569 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.583522 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e55b0a1-21ae-406d-af93-b1846b5e891f-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:48:44.583747 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.583599 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9z6c\" (UniqueName: \"kubernetes.io/projected/9e55b0a1-21ae-406d-af93-b1846b5e891f-kube-api-access-x9z6c\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:48:44.592253 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.592215 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:44.627183 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.627151 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92" Oct 30 01:48:44.930788 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:44.930738 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92"] Oct 30 01:48:44.935914 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:48:44.935885 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2ae791_7e6e_493c_8927_7116f567eb24.slice/crio-48eb6ef412a49a875f9cff07b06cfcb3a84c18599446c7d7b1f9d57fc3e8b945 WatchSource:0}: Error finding container 48eb6ef412a49a875f9cff07b06cfcb3a84c18599446c7d7b1f9d57fc3e8b945: Status 404 returned error can't find the container with id 48eb6ef412a49a875f9cff07b06cfcb3a84c18599446c7d7b1f9d57fc3e8b945 Oct 30 01:48:45.280461 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:45.280383 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf" Oct 30 01:48:45.280461 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:45.280421 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf" event={"ID":"9e55b0a1-21ae-406d-af93-b1846b5e891f","Type":"ContainerDied","Data":"1025d89048bb02fbd0eb015d25e6533b41ab5527fec9938382b56998465f82ce"} Oct 30 01:48:45.280655 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:45.280480 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1025d89048bb02fbd0eb015d25e6533b41ab5527fec9938382b56998465f82ce" Oct 30 01:48:45.282128 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:45.282075 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92" event={"ID":"9e2ae791-7e6e-493c-8927-7116f567eb24","Type":"ContainerStarted","Data":"0d49e19040689eb87933d15c4e47dba469c1f26905f0459607ccd3c9915921e7"} Oct 30 01:48:45.282249 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:45.282152 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92" event={"ID":"9e2ae791-7e6e-493c-8927-7116f567eb24","Type":"ContainerStarted","Data":"48eb6ef412a49a875f9cff07b06cfcb3a84c18599446c7d7b1f9d57fc3e8b945"} Oct 30 01:48:45.321789 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:45.321730 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92" podStartSLOduration=1.321714953 podStartE2EDuration="1.321714953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:45.320942867 +0000 UTC m=+1154.899733949" watchObservedRunningTime="2026-10-30 01:48:45.321714953 +0000 UTC m=+1154.900506033" Oct 30 01:48:45.333863 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:45.333829 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:46.160637 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.160594 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg"] Oct 30 01:48:46.161228 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.161209 5998 cpu_manager.go:414] "containerMap: removing container" podUID="9e55b0a1-21ae-406d-af93-b1846b5e891f" containerName="revoke-keys" Oct 30 01:48:46.161277 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.161231 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e55b0a1-21ae-406d-af93-b1846b5e891f" containerName="revoke-keys" Oct 30 01:48:46.161325 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.161314 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e55b0a1-21ae-406d-af93-b1846b5e891f" containerName="revoke-keys" Oct 30 01:48:46.164993 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.164975 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg" Oct 30 01:48:46.190903 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.190876 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg"] Oct 30 01:48:46.197853 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.197817 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg\" (UID: \"2a1a6ba0-9516-46c5-95f4-8cfdc7114266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg" Oct 30 01:48:46.198595 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.197874 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg\" (UID: \"2a1a6ba0-9516-46c5-95f4-8cfdc7114266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg" Oct 30 01:48:46.198595 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.197982 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pns4\" (UniqueName: \"kubernetes.io/projected/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-kube-api-access-9pns4\") pod \"maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg\" (UID: \"2a1a6ba0-9516-46c5-95f4-8cfdc7114266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg" Oct 30 01:48:46.289123 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.289068 5998 generic.go:356] "Generic (PLEG): container finished" podID="9e2ae791-7e6e-493c-8927-7116f567eb24" containerID="0d49e19040689eb87933d15c4e47dba469c1f26905f0459607ccd3c9915921e7" exitCode=0 Oct 30 01:48:46.289123 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.289122 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92" event={"ID":"9e2ae791-7e6e-493c-8927-7116f567eb24","Type":"ContainerDied","Data":"0d49e19040689eb87933d15c4e47dba469c1f26905f0459607ccd3c9915921e7"} Oct 30 01:48:46.298616 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.298587 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pns4\" (UniqueName: \"kubernetes.io/projected/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-kube-api-access-9pns4\") pod \"maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg\" (UID: \"2a1a6ba0-9516-46c5-95f4-8cfdc7114266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg" Oct 30 01:48:46.298774 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.298673 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg\" (UID: \"2a1a6ba0-9516-46c5-95f4-8cfdc7114266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg" Oct 30 01:48:46.298774 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.298705 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg\" (UID: \"2a1a6ba0-9516-46c5-95f4-8cfdc7114266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg" Oct 30 01:48:46.299341 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.299318 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg\" (UID: \"2a1a6ba0-9516-46c5-95f4-8cfdc7114266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg" Oct 30 01:48:46.301199 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.301181 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg\" (UID: \"2a1a6ba0-9516-46c5-95f4-8cfdc7114266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg" Oct 30 01:48:46.323699 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.323667 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pns4\" (UniqueName: \"kubernetes.io/projected/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-kube-api-access-9pns4\") pod \"maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg\" (UID: \"2a1a6ba0-9516-46c5-95f4-8cfdc7114266\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg" Oct 30 01:48:46.478679 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.478599 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg" Oct 30 01:48:46.805626 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:46.805541 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg"] Oct 30 01:48:46.810330 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:48:46.810285 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1a6ba0_9516_46c5_95f4_8cfdc7114266.slice/crio-a601b13326c30be256f1588243d4c09334fd1bdd7bebac4bb91f13d99d35ff56 WatchSource:0}: Error finding container a601b13326c30be256f1588243d4c09334fd1bdd7bebac4bb91f13d99d35ff56: Status 404 returned error can't find the container with id a601b13326c30be256f1588243d4c09334fd1bdd7bebac4bb91f13d99d35ff56 Oct 30 01:48:47.295779 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:47.295740 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg" event={"ID":"2a1a6ba0-9516-46c5-95f4-8cfdc7114266","Type":"ContainerStarted","Data":"e207db18a1c3f4b2d81ee1425af7812dee59380ff21f1ad169da49df339de5e7"} Oct 30 01:48:47.295989 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:47.295805 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg" event={"ID":"2a1a6ba0-9516-46c5-95f4-8cfdc7114266","Type":"ContainerStarted","Data":"a601b13326c30be256f1588243d4c09334fd1bdd7bebac4bb91f13d99d35ff56"} Oct 30 01:48:47.336325 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:47.336263 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg" podStartSLOduration=1.336246515 podStartE2EDuration="1.336246515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:47.33520233 +0000 UTC m=+1156.913993411" watchObservedRunningTime="2026-10-30 01:48:47.336246515 +0000 UTC m=+1156.915037596" Oct 30 01:48:47.353671 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:47.353642 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:47.442739 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:47.442706 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92" Oct 30 01:48:47.511520 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:47.511487 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e2ae791-7e6e-493c-8927-7116f567eb24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2ae791-7e6e-493c-8927-7116f567eb24-maas-api-xks-ca\") pod \"9e2ae791-7e6e-493c-8927-7116f567eb24\" (UID: \"9e2ae791-7e6e-493c-8927-7116f567eb24\") " Oct 30 01:48:47.511712 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:47.511604 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e2ae791-7e6e-493c-8927-7116f567eb24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2ae791-7e6e-493c-8927-7116f567eb24-maas-api-openshift-ca\") pod \"9e2ae791-7e6e-493c-8927-7116f567eb24\" (UID: \"9e2ae791-7e6e-493c-8927-7116f567eb24\") " Oct 30 01:48:47.511712 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:47.511702 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e2ae791-7e6e-493c-8927-7116f567eb24-kube-api-access-rmps6\" (UniqueName: \"kubernetes.io/projected/9e2ae791-7e6e-493c-8927-7116f567eb24-kube-api-access-rmps6\") pod \"9e2ae791-7e6e-493c-8927-7116f567eb24\" (UID: \"9e2ae791-7e6e-493c-8927-7116f567eb24\") " Oct 30 01:48:47.512028 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:47.511997 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e2ae791-7e6e-493c-8927-7116f567eb24-maas-api-openshift-ca" pod "9e2ae791-7e6e-493c-8927-7116f567eb24" (UID: "9e2ae791-7e6e-493c-8927-7116f567eb24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:48:47.513838 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:47.513808 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e2ae791-7e6e-493c-8927-7116f567eb24-maas-api-xks-ca" pod "9e2ae791-7e6e-493c-8927-7116f567eb24" (UID: "9e2ae791-7e6e-493c-8927-7116f567eb24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:48:47.514302 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:47.514278 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2ae791-7e6e-493c-8927-7116f567eb24-kube-api-access-rmps6" pod "9e2ae791-7e6e-493c-8927-7116f567eb24" (UID: "9e2ae791-7e6e-493c-8927-7116f567eb24"). InnerVolumeSpecName "kube-api-access-rmps6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:48:47.612928 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:47.612877 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2ae791-7e6e-493c-8927-7116f567eb24-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:48:47.612928 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:47.612922 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmps6\" (UniqueName: \"kubernetes.io/projected/9e2ae791-7e6e-493c-8927-7116f567eb24-kube-api-access-rmps6\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:48:47.613217 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:47.612939 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2ae791-7e6e-493c-8927-7116f567eb24-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:48:48.302449 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:48.302414 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92" Oct 30 01:48:48.302660 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:48.302455 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92" event={"ID":"9e2ae791-7e6e-493c-8927-7116f567eb24","Type":"ContainerDied","Data":"48eb6ef412a49a875f9cff07b06cfcb3a84c18599446c7d7b1f9d57fc3e8b945"} Oct 30 01:48:48.302660 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:48.302497 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48eb6ef412a49a875f9cff07b06cfcb3a84c18599446c7d7b1f9d57fc3e8b945" Oct 30 01:48:48.304056 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:48.304032 5998 generic.go:356] "Generic (PLEG): container finished" podID="2a1a6ba0-9516-46c5-95f4-8cfdc7114266" containerID="e207db18a1c3f4b2d81ee1425af7812dee59380ff21f1ad169da49df339de5e7" exitCode=0 Oct 30 01:48:48.304157 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:48.304081 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg" event={"ID":"2a1a6ba0-9516-46c5-95f4-8cfdc7114266","Type":"ContainerDied","Data":"e207db18a1c3f4b2d81ee1425af7812dee59380ff21f1ad169da49df339de5e7"} Oct 30 01:48:49.135885 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:49.135849 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:49.441550 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:49.441523 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg" Oct 30 01:48:49.531141 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:49.531075 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-maas-api-openshift-ca\") pod \"2a1a6ba0-9516-46c5-95f4-8cfdc7114266\" (UID: \"2a1a6ba0-9516-46c5-95f4-8cfdc7114266\") " Oct 30 01:48:49.531314 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:49.531187 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-kube-api-access-9pns4\" (UniqueName: \"kubernetes.io/projected/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-kube-api-access-9pns4\") pod \"2a1a6ba0-9516-46c5-95f4-8cfdc7114266\" (UID: \"2a1a6ba0-9516-46c5-95f4-8cfdc7114266\") " Oct 30 01:48:49.531314 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:49.531220 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-maas-api-xks-ca\") pod \"2a1a6ba0-9516-46c5-95f4-8cfdc7114266\" (UID: \"2a1a6ba0-9516-46c5-95f4-8cfdc7114266\") " Oct 30 01:48:49.531496 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:49.531467 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-maas-api-openshift-ca" pod "2a1a6ba0-9516-46c5-95f4-8cfdc7114266" (UID: "2a1a6ba0-9516-46c5-95f4-8cfdc7114266"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:48:49.531612 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:49.531589 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:48:49.533290 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:49.533266 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-kube-api-access-9pns4" pod "2a1a6ba0-9516-46c5-95f4-8cfdc7114266" (UID: "2a1a6ba0-9516-46c5-95f4-8cfdc7114266"). InnerVolumeSpecName "kube-api-access-9pns4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:48:49.533388 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:49.533289 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-maas-api-xks-ca" pod "2a1a6ba0-9516-46c5-95f4-8cfdc7114266" (UID: "2a1a6ba0-9516-46c5-95f4-8cfdc7114266"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:48:49.632910 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:49.632874 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pns4\" (UniqueName: \"kubernetes.io/projected/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-kube-api-access-9pns4\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:48:49.632910 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:49.632904 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a1a6ba0-9516-46c5-95f4-8cfdc7114266-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:48:49.711906 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:49.711826 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:49.820461 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:49.820423 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:50.315805 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:50.315778 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg" Oct 30 01:48:50.316267 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:50.315809 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg" event={"ID":"2a1a6ba0-9516-46c5-95f4-8cfdc7114266","Type":"ContainerDied","Data":"a601b13326c30be256f1588243d4c09334fd1bdd7bebac4bb91f13d99d35ff56"} Oct 30 01:48:50.316267 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:50.315891 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a601b13326c30be256f1588243d4c09334fd1bdd7bebac4bb91f13d99d35ff56" Oct 30 01:48:50.890391 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:50.890353 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:52.255518 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:52.255485 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:52.622656 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:52.622566 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:52.733223 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:52.733191 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:52.840933 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:52.840873 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:53.016750 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:53.016708 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:53.211974 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:53.211941 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:53.325858 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:53.325758 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:53.329610 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:53.329584 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf"] Oct 30 01:48:53.341307 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:53.341266 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7664a237-2b2mf"] Oct 30 01:48:53.520476 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:53.520426 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:53.824026 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:53.823983 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:54.296125 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:54.296076 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:54.408698 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:54.408669 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:54.517389 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:54.517358 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:54.537914 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:54.537883 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e55b0a1-21ae-406d-af93-b1846b5e891f" path="/var/lib/kubelet/pods/9e55b0a1-21ae-406d-af93-b1846b5e891f/volumes" Oct 30 01:48:54.869223 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:54.869192 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:55.017067 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:55.016510 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:55.142125 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:55.142037 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:55.216449 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:48:55.216401 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:48:55.623138 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:55.623098 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:56.002905 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:56.002866 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:48:56.276252 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:56.276166 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:56.389045 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:56.389003 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:56.422122 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:56.422046 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92"] Oct 30 01:48:56.433800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:56.433769 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-61b6469a-5ds92"] Oct 30 01:48:56.538499 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:56.538412 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e2ae791-7e6e-493c-8927-7116f567eb24" path="/var/lib/kubelet/pods/9e2ae791-7e6e-493c-8927-7116f567eb24/volumes" Oct 30 01:48:57.057183 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:57.057137 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:57.196444 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:57.196406 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:57.808467 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:57.808435 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:57.925982 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:57.925947 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:58.199997 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.199962 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:58.318555 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.318523 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:58.413604 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.413567 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz"] Oct 30 01:48:58.414154 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.414125 5998 cpu_manager.go:414] "containerMap: removing container" podUID="9e2ae791-7e6e-493c-8927-7116f567eb24" containerName="revoke-keys" Oct 30 01:48:58.414154 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.414152 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e2ae791-7e6e-493c-8927-7116f567eb24" containerName="revoke-keys" Oct 30 01:48:58.414339 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.414196 5998 cpu_manager.go:414] "containerMap: removing container" podUID="2a1a6ba0-9516-46c5-95f4-8cfdc7114266" containerName="revoke-keys" Oct 30 01:48:58.414339 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.414203 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1a6ba0-9516-46c5-95f4-8cfdc7114266" containerName="revoke-keys" Oct 30 01:48:58.414339 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.414303 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1a6ba0-9516-46c5-95f4-8cfdc7114266" containerName="revoke-keys" Oct 30 01:48:58.414339 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.414319 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e2ae791-7e6e-493c-8927-7116f567eb24" containerName="revoke-keys" Oct 30 01:48:58.418551 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.418531 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz" Oct 30 01:48:58.444946 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.444919 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:48:58.444946 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.444945 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:48:58.447365 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.447340 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz"] Oct 30 01:48:58.454664 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.454613 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:48:58.454664 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.454613 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:48:58.516485 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.516443 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcnl4\" (UniqueName: \"kubernetes.io/projected/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-kube-api-access-gcnl4\") pod \"maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz\" (UID: \"b4b5eeb8-6fc7-4f90-90c1-da67fb640606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz" Oct 30 01:48:58.516706 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.516506 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz\" (UID: \"b4b5eeb8-6fc7-4f90-90c1-da67fb640606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz" Oct 30 01:48:58.516706 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.516675 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz\" (UID: \"b4b5eeb8-6fc7-4f90-90c1-da67fb640606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz" Oct 30 01:48:58.571934 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.571904 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:58.617324 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.617285 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz\" (UID: \"b4b5eeb8-6fc7-4f90-90c1-da67fb640606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz" Oct 30 01:48:58.617497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.617385 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz\" (UID: \"b4b5eeb8-6fc7-4f90-90c1-da67fb640606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz" Oct 30 01:48:58.617497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.617417 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcnl4\" (UniqueName: \"kubernetes.io/projected/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-kube-api-access-gcnl4\") pod \"maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz\" (UID: \"b4b5eeb8-6fc7-4f90-90c1-da67fb640606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz" Oct 30 01:48:58.618161 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.618136 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz\" (UID: \"b4b5eeb8-6fc7-4f90-90c1-da67fb640606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz" Oct 30 01:48:58.619772 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.619741 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz\" (UID: \"b4b5eeb8-6fc7-4f90-90c1-da67fb640606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz" Oct 30 01:48:58.651511 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.651479 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcnl4\" (UniqueName: \"kubernetes.io/projected/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-kube-api-access-gcnl4\") pod \"maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz\" (UID: \"b4b5eeb8-6fc7-4f90-90c1-da67fb640606\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz" Oct 30 01:48:58.716957 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.716840 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:58.731269 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.731239 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz" Oct 30 01:48:58.842687 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:58.842655 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:59.043116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:59.042989 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz"] Oct 30 01:48:59.048132 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:48:59.048074 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b5eeb8_6fc7_4f90_90c1_da67fb640606.slice/crio-c64b6506350be7e021908c7e2c65da0f2492d41ecbfa115c8a516abe47323676 WatchSource:0}: Error finding container c64b6506350be7e021908c7e2c65da0f2492d41ecbfa115c8a516abe47323676: Status 404 returned error can't find the container with id c64b6506350be7e021908c7e2c65da0f2492d41ecbfa115c8a516abe47323676 Oct 30 01:48:59.370918 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:59.370881 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz" event={"ID":"b4b5eeb8-6fc7-4f90-90c1-da67fb640606","Type":"ContainerStarted","Data":"c76d3be1c8b1c0aa31c76d7b6c72482d2804d1517ae1a09ead38eb78f2fd1f8e"} Oct 30 01:48:59.371400 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:59.371195 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz" event={"ID":"b4b5eeb8-6fc7-4f90-90c1-da67fb640606","Type":"ContainerStarted","Data":"c64b6506350be7e021908c7e2c65da0f2492d41ecbfa115c8a516abe47323676"} Oct 30 01:48:59.406996 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:59.406961 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:59.438274 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:59.436474 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz" podStartSLOduration=1.43645355 podStartE2EDuration="1.43645355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:59.434577386 +0000 UTC m=+1169.013368494" watchObservedRunningTime="2026-10-30 01:48:59.43645355 +0000 UTC m=+1169.015244632" Oct 30 01:48:59.547213 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:59.547178 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:48:59.606258 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:59.606115 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg"] Oct 30 01:48:59.610586 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:59.610550 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-14b53b75-x5fjg"] Oct 30 01:48:59.663186 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:48:59.663152 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:00.040514 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:00.040478 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:00.261403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:00.261327 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:00.371448 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:00.371395 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:00.377953 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:00.377920 5998 generic.go:356] "Generic (PLEG): container finished" podID="b4b5eeb8-6fc7-4f90-90c1-da67fb640606" containerID="c76d3be1c8b1c0aa31c76d7b6c72482d2804d1517ae1a09ead38eb78f2fd1f8e" exitCode=0 Oct 30 01:49:00.378080 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:00.377999 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz" event={"ID":"b4b5eeb8-6fc7-4f90-90c1-da67fb640606","Type":"ContainerDied","Data":"c76d3be1c8b1c0aa31c76d7b6c72482d2804d1517ae1a09ead38eb78f2fd1f8e"} Oct 30 01:49:00.544178 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:00.544083 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1a6ba0-9516-46c5-95f4-8cfdc7114266" path="/var/lib/kubelet/pods/2a1a6ba0-9516-46c5-95f4-8cfdc7114266/volumes" Oct 30 01:49:01.006780 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:01.006741 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:01.474986 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:01.474153 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:01.538388 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:01.538360 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz" Oct 30 01:49:01.591807 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:01.591769 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:01.651138 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:01.651006 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-maas-api-openshift-ca\") pod \"b4b5eeb8-6fc7-4f90-90c1-da67fb640606\" (UID: \"b4b5eeb8-6fc7-4f90-90c1-da67fb640606\") " Oct 30 01:49:01.651138 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:01.651076 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-kube-api-access-gcnl4\" (UniqueName: \"kubernetes.io/projected/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-kube-api-access-gcnl4\") pod \"b4b5eeb8-6fc7-4f90-90c1-da67fb640606\" (UID: \"b4b5eeb8-6fc7-4f90-90c1-da67fb640606\") " Oct 30 01:49:01.651374 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:01.651144 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-maas-api-xks-ca\") pod \"b4b5eeb8-6fc7-4f90-90c1-da67fb640606\" (UID: \"b4b5eeb8-6fc7-4f90-90c1-da67fb640606\") " Oct 30 01:49:01.651554 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:01.651497 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-maas-api-openshift-ca" pod "b4b5eeb8-6fc7-4f90-90c1-da67fb640606" (UID: "b4b5eeb8-6fc7-4f90-90c1-da67fb640606"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:49:01.651870 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:01.651830 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:49:01.653548 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:01.653521 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-kube-api-access-gcnl4" pod "b4b5eeb8-6fc7-4f90-90c1-da67fb640606" (UID: "b4b5eeb8-6fc7-4f90-90c1-da67fb640606"). InnerVolumeSpecName "kube-api-access-gcnl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:01.653771 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:01.653751 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-maas-api-xks-ca" pod "b4b5eeb8-6fc7-4f90-90c1-da67fb640606" (UID: "b4b5eeb8-6fc7-4f90-90c1-da67fb640606"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:49:01.752866 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:01.752825 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcnl4\" (UniqueName: \"kubernetes.io/projected/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-kube-api-access-gcnl4\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:49:01.752866 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:01.752862 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4b5eeb8-6fc7-4f90-90c1-da67fb640606-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:49:01.941172 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:01.941138 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:02.066444 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:02.066401 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:02.229573 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:02.229488 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:02.354245 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:02.354209 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:02.391569 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:02.391535 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz" event={"ID":"b4b5eeb8-6fc7-4f90-90c1-da67fb640606","Type":"ContainerDied","Data":"c64b6506350be7e021908c7e2c65da0f2492d41ecbfa115c8a516abe47323676"} Oct 30 01:49:02.391569 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:02.391563 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz" Oct 30 01:49:02.391776 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:02.391588 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c64b6506350be7e021908c7e2c65da0f2492d41ecbfa115c8a516abe47323676" Oct 30 01:49:03.374551 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:03.374513 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:03.494216 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:03.494176 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:03.940773 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:03.940740 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:04.135623 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:04.135589 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:05.425754 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:05.425715 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:05.536517 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:05.536486 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:05.689541 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:05.689443 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:49:05.941670 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:05.941593 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:06.592412 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:06.592370 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:06.706875 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:06.706837 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:07.498284 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:07.498242 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:07.611473 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:07.611427 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:08.098954 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:08.098922 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:08.831216 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:08.831177 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:08.938897 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:08.938851 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:09.517278 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:09.517247 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:49:09.664404 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:09.664361 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:09.773056 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:09.772968 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:11.387132 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:11.387101 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:13.012154 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:13.012064 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:14.873727 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:14.873691 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:14.986540 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:14.986507 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:16.809386 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:16.809346 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:18.101388 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:18.101346 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:18.221877 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:18.221834 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:18.327804 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:18.327771 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:18.690998 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:18.690965 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:18.789002 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:18.788963 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:49:18.889488 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:18.889448 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz"] Oct 30 01:49:18.892503 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:18.892476 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-6c3505dc-xpvmz"] Oct 30 01:49:19.532415 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:19.532380 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:19.856689 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:19.856603 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:20.512315 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:20.512276 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:20.544683 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:20.544641 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4b5eeb8-6fc7-4f90-90c1-da67fb640606" path="/var/lib/kubelet/pods/b4b5eeb8-6fc7-4f90-90c1-da67fb640606/volumes" Oct 30 01:49:20.663783 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:20.663720 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:21.111954 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:21.111917 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:22.191793 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.191758 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx"] Oct 30 01:49:22.192351 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.192330 5998 cpu_manager.go:414] "containerMap: removing container" podUID="b4b5eeb8-6fc7-4f90-90c1-da67fb640606" containerName="revoke-keys" Oct 30 01:49:22.192418 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.192352 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4b5eeb8-6fc7-4f90-90c1-da67fb640606" containerName="revoke-keys" Oct 30 01:49:22.192475 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.192443 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4b5eeb8-6fc7-4f90-90c1-da67fb640606" containerName="revoke-keys" Oct 30 01:49:22.195739 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.195711 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.230705 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.230677 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-78d5ff\"/\"default-dockercfg-clrdr\"" Oct 30 01:49:22.230705 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.230678 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-78d5ff\"/\"simulator-w3-78d5ff-kserve-self-signed-certs\"" Oct 30 01:49:22.230953 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.230718 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-78d5ff\"/\"openshift-service-ca.crt\"" Oct 30 01:49:22.230953 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.230912 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-78d5ff\"/\"kube-root-ca.crt\"" Oct 30 01:49:22.232766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.232740 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx"] Oct 30 01:49:22.277112 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.277045 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:22.344464 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.344421 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-tmp-dir\") pod \"simulator-w3-78d5ff-kserve-9c499455d-vkgtx\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.344663 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.344485 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-model-cache\") pod \"simulator-w3-78d5ff-kserve-9c499455d-vkgtx\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.344663 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.344574 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-home\") pod \"simulator-w3-78d5ff-kserve-9c499455d-vkgtx\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.344663 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.344601 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4lk6\" (UniqueName: \"kubernetes.io/projected/bae44a76-84eb-475f-abea-de3828e1f033-kube-api-access-w4lk6\") pod \"simulator-w3-78d5ff-kserve-9c499455d-vkgtx\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.344663 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.344632 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-dshm\") pod \"simulator-w3-78d5ff-kserve-9c499455d-vkgtx\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.344886 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.344784 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bae44a76-84eb-475f-abea-de3828e1f033-tls-certs\") pod \"simulator-w3-78d5ff-kserve-9c499455d-vkgtx\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.445897 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.445798 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-tmp-dir\") pod \"simulator-w3-78d5ff-kserve-9c499455d-vkgtx\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.446219 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.446185 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-model-cache\") pod \"simulator-w3-78d5ff-kserve-9c499455d-vkgtx\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.446351 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.446229 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-home\") pod \"simulator-w3-78d5ff-kserve-9c499455d-vkgtx\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.446351 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.446256 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4lk6\" (UniqueName: \"kubernetes.io/projected/bae44a76-84eb-475f-abea-de3828e1f033-kube-api-access-w4lk6\") pod \"simulator-w3-78d5ff-kserve-9c499455d-vkgtx\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.446351 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.446286 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-dshm\") pod \"simulator-w3-78d5ff-kserve-9c499455d-vkgtx\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.446351 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.446320 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-tmp-dir\") pod \"simulator-w3-78d5ff-kserve-9c499455d-vkgtx\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.446531 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.446381 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bae44a76-84eb-475f-abea-de3828e1f033-tls-certs\") pod \"simulator-w3-78d5ff-kserve-9c499455d-vkgtx\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.446942 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.446914 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-model-cache\") pod \"simulator-w3-78d5ff-kserve-9c499455d-vkgtx\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.447021 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.446984 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-home\") pod \"simulator-w3-78d5ff-kserve-9c499455d-vkgtx\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.448770 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.448749 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-dshm\") pod \"simulator-w3-78d5ff-kserve-9c499455d-vkgtx\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.449247 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.449213 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bae44a76-84eb-475f-abea-de3828e1f033-tls-certs\") pod \"simulator-w3-78d5ff-kserve-9c499455d-vkgtx\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.469341 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.469308 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4lk6\" (UniqueName: \"kubernetes.io/projected/bae44a76-84eb-475f-abea-de3828e1f033-kube-api-access-w4lk6\") pod \"simulator-w3-78d5ff-kserve-9c499455d-vkgtx\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.508897 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.508864 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:22.816421 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:22.816301 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx"] Oct 30 01:49:22.821723 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:49:22.821695 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbae44a76_84eb_475f_abea_de3828e1f033.slice/crio-107db43a7c1776e4facbda5322416f94c5fe03bf35ce09ac31e38e1713dd4808 WatchSource:0}: Error finding container 107db43a7c1776e4facbda5322416f94c5fe03bf35ce09ac31e38e1713dd4808: Status 404 returned error can't find the container with id 107db43a7c1776e4facbda5322416f94c5fe03bf35ce09ac31e38e1713dd4808 Oct 30 01:49:23.177586 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:23.177544 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:23.289227 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:23.289189 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:23.523209 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:23.523118 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" event={"ID":"bae44a76-84eb-475f-abea-de3828e1f033","Type":"ContainerStarted","Data":"107db43a7c1776e4facbda5322416f94c5fe03bf35ce09ac31e38e1713dd4808"} Oct 30 01:49:23.788585 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:23.788491 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:23.900641 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:23.900605 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:24.008314 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:24.008282 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:24.537649 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:24.537614 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" event={"ID":"bae44a76-84eb-475f-abea-de3828e1f033","Type":"ContainerStarted","Data":"ed3802a5221cffbad50df98efa8690453f3ac1ebd17d9c35915028ead3609c1d"} Oct 30 01:49:24.572531 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:24.572479 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podStartSLOduration=1.425336289 podStartE2EDuration="2.572467011s" totalImagesPullingTime="1.147130722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:24.571431158 +0000 UTC m=+1194.150222238" watchObservedRunningTime="2026-10-30 01:49:24.572467011 +0000 UTC m=+1194.151258091" Oct 30 01:49:24.812359 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:24.812267 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:24.920302 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:24.920269 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:24.993230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:24.993193 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:49:25.217296 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:49:25.217254 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:49:25.435744 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:25.435712 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:25.509621 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:25.509539 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:25.558197 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:25.558150 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:26.490070 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:26.490033 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:26.606278 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:26.606246 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:27.479239 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:27.479204 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:29.287493 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:29.287457 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:30.439711 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:30.439675 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:31.023271 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:31.023230 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:31.141397 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:31.141366 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:31.249557 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:31.249519 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:32.509149 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:32.509112 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:32.521920 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:32.521889 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:32.584711 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:32.584679 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:49:33.079108 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:33.079052 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:33.673898 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:33.673862 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:49:33.814341 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:33.814310 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:34.993497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:34.993465 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:35.100556 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.100518 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:35.245232 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.245144 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:35.655931 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.655847 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv"] Oct 30 01:49:35.660934 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.660905 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.675698 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.675595 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b93d51\"/\"openshift-service-ca.crt\"" Oct 30 01:49:35.677990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.677962 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-home\") pod \"simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.678125 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.678023 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhcx4\" (UniqueName: \"kubernetes.io/projected/e4786840-065b-49eb-854c-87ecd6b08916-kube-api-access-vhcx4\") pod \"simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.678225 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.678198 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4786840-065b-49eb-854c-87ecd6b08916-tls-certs\") pod \"simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.678283 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.678267 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-dshm\") pod \"simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.678399 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.678374 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-model-cache\") pod \"simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.678453 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.678414 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-tmp-dir\") pod \"simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.687204 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.687170 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b93d51\"/\"kube-root-ca.crt\"" Oct 30 01:49:35.687361 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.687225 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b93d51\"/\"simulator-w4-b93d51-kserve-self-signed-certs\"" Oct 30 01:49:35.687361 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.687346 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b93d51\"/\"default-dockercfg-cj96z\"" Oct 30 01:49:35.689295 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.689272 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv"] Oct 30 01:49:35.707511 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.707479 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:35.779681 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.779644 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-model-cache\") pod \"simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.779681 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.779687 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-tmp-dir\") pod \"simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.779945 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.779809 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-home\") pod \"simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.779945 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.779871 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhcx4\" (UniqueName: \"kubernetes.io/projected/e4786840-065b-49eb-854c-87ecd6b08916-kube-api-access-vhcx4\") pod \"simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.780053 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.779941 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4786840-065b-49eb-854c-87ecd6b08916-tls-certs\") pod \"simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.780053 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.779997 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-dshm\") pod \"simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.780182 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.780166 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-model-cache\") pod \"simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.780344 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.780320 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-tmp-dir\") pod \"simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.780457 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.780437 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-home\") pod \"simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.782483 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.782459 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-dshm\") pod \"simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.782739 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.782721 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4786840-065b-49eb-854c-87ecd6b08916-tls-certs\") pod \"simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.803135 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.803076 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhcx4\" (UniqueName: \"kubernetes.io/projected/e4786840-065b-49eb-854c-87ecd6b08916-kube-api-access-vhcx4\") pod \"simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:35.976754 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:35.976708 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:36.279315 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:36.279214 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv"] Oct 30 01:49:36.284510 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:49:36.284477 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4786840_065b_49eb_854c_87ecd6b08916.slice/crio-dec74e26eca41dd1368a729ca3526f1470bbf69883125c82eb763d839f0b1f9e WatchSource:0}: Error finding container dec74e26eca41dd1368a729ca3526f1470bbf69883125c82eb763d839f0b1f9e: Status 404 returned error can't find the container with id dec74e26eca41dd1368a729ca3526f1470bbf69883125c82eb763d839f0b1f9e Oct 30 01:49:36.597352 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:36.597265 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" event={"ID":"e4786840-065b-49eb-854c-87ecd6b08916","Type":"ContainerStarted","Data":"e465c9888d7256a975fce10ef6bbdca209528c2d3a1fbeb5104578c317116b19"} Oct 30 01:49:36.597352 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:36.597319 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" event={"ID":"e4786840-065b-49eb-854c-87ecd6b08916","Type":"ContainerStarted","Data":"dec74e26eca41dd1368a729ca3526f1470bbf69883125c82eb763d839f0b1f9e"} Oct 30 01:49:36.638112 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:36.638035 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podStartSLOduration=1.638020273 podStartE2EDuration="1.638020273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:36.637391667 +0000 UTC m=+1206.216182748" watchObservedRunningTime="2026-10-30 01:49:36.638020273 +0000 UTC m=+1206.216811354" Oct 30 01:49:36.897755 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:36.897681 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:36.976960 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:36.976925 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:38.495810 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.495780 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:38.757594 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.757506 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p"] Oct 30 01:49:38.761833 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.761796 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:38.777024 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.777001 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c40a89\"/\"kube-root-ca.crt\"" Oct 30 01:49:38.788540 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.788515 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c40a89\"/\"openshift-service-ca.crt\"" Oct 30 01:49:38.788691 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.788583 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c40a89\"/\"simulator-w0-c40a89-kserve-self-signed-certs\"" Oct 30 01:49:38.788691 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.788606 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c40a89\"/\"default-dockercfg-rp6jt\"" Oct 30 01:49:38.791530 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.791506 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p"] Oct 30 01:49:38.807267 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.807235 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:38.807590 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.807563 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-dshm\") pod \"simulator-w0-c40a89-kserve-6db79f49b6-xq52p\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:38.807703 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.807613 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spntn\" (UniqueName: \"kubernetes.io/projected/a7c758e0-df24-4751-9751-24d3050bdcbb-kube-api-access-spntn\") pod \"simulator-w0-c40a89-kserve-6db79f49b6-xq52p\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:38.807747 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.807728 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7c758e0-df24-4751-9751-24d3050bdcbb-tls-certs\") pod \"simulator-w0-c40a89-kserve-6db79f49b6-xq52p\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:38.807825 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.807799 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-home\") pod \"simulator-w0-c40a89-kserve-6db79f49b6-xq52p\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:38.807860 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.807843 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-model-cache\") pod \"simulator-w0-c40a89-kserve-6db79f49b6-xq52p\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:38.807967 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.807945 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-tmp-dir\") pod \"simulator-w0-c40a89-kserve-6db79f49b6-xq52p\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:38.910674 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.910625 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7c758e0-df24-4751-9751-24d3050bdcbb-tls-certs\") pod \"simulator-w0-c40a89-kserve-6db79f49b6-xq52p\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:38.910865 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.910727 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-home\") pod \"simulator-w0-c40a89-kserve-6db79f49b6-xq52p\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:38.910865 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.910758 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-model-cache\") pod \"simulator-w0-c40a89-kserve-6db79f49b6-xq52p\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:38.910865 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.910802 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-tmp-dir\") pod \"simulator-w0-c40a89-kserve-6db79f49b6-xq52p\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:38.910865 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.910861 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-dshm\") pod \"simulator-w0-c40a89-kserve-6db79f49b6-xq52p\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:38.911119 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.910894 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spntn\" (UniqueName: \"kubernetes.io/projected/a7c758e0-df24-4751-9751-24d3050bdcbb-kube-api-access-spntn\") pod \"simulator-w0-c40a89-kserve-6db79f49b6-xq52p\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:38.911544 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.911519 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-model-cache\") pod \"simulator-w0-c40a89-kserve-6db79f49b6-xq52p\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:38.911624 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.911563 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-tmp-dir\") pod \"simulator-w0-c40a89-kserve-6db79f49b6-xq52p\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:38.911789 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.911767 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-home\") pod \"simulator-w0-c40a89-kserve-6db79f49b6-xq52p\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:38.914561 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.914537 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7c758e0-df24-4751-9751-24d3050bdcbb-tls-certs\") pod \"simulator-w0-c40a89-kserve-6db79f49b6-xq52p\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:38.915319 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.915292 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-dshm\") pod \"simulator-w0-c40a89-kserve-6db79f49b6-xq52p\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:38.934587 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:38.934558 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spntn\" (UniqueName: \"kubernetes.io/projected/a7c758e0-df24-4751-9751-24d3050bdcbb-kube-api-access-spntn\") pod \"simulator-w0-c40a89-kserve-6db79f49b6-xq52p\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:39.074336 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:39.074251 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:39.389116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:39.388998 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p"] Oct 30 01:49:39.393707 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:49:39.393665 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7c758e0_df24_4751_9751_24d3050bdcbb.slice/crio-7247185310bb83e1b93f411c844e647e34e1531d26f4d250c59df7ea1b55a6fc WatchSource:0}: Error finding container 7247185310bb83e1b93f411c844e647e34e1531d26f4d250c59df7ea1b55a6fc: Status 404 returned error can't find the container with id 7247185310bb83e1b93f411c844e647e34e1531d26f4d250c59df7ea1b55a6fc Oct 30 01:49:39.614729 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:39.614692 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" event={"ID":"a7c758e0-df24-4751-9751-24d3050bdcbb","Type":"ContainerStarted","Data":"fda30aa28d3fe47f53a954d9d2fb10c24d79501d5445b90b793786157ad4f2e1"} Oct 30 01:49:39.614729 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:39.614734 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" event={"ID":"a7c758e0-df24-4751-9751-24d3050bdcbb","Type":"ContainerStarted","Data":"7247185310bb83e1b93f411c844e647e34e1531d26f4d250c59df7ea1b55a6fc"} Oct 30 01:49:39.655992 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:39.655890 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podStartSLOduration=1.655875704 podStartE2EDuration="1.655875704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:39.654631273 +0000 UTC m=+1209.233422355" watchObservedRunningTime="2026-10-30 01:49:39.655875704 +0000 UTC m=+1209.234666784" Oct 30 01:49:40.074410 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:40.074356 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:40.210507 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:40.210470 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:41.858555 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:41.858520 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:43.364020 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:43.363991 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:44.957536 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:44.957504 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:45.486691 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:45.486648 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:49:45.977487 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:45.977448 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:45.990954 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:45.990921 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:46.565197 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:46.565165 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:46.666062 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:46.666032 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:49:48.276144 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:48.276105 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:49.074775 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:49.074739 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:49.088142 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:49.088112 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:49.683259 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:49.683226 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:49:49.888118 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:49.888070 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:50.080148 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:50.080059 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:50.652912 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:50.652879 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:50.765558 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:50.765525 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:50.889270 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:50.889239 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:51.132868 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.132830 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw"] Oct 30 01:49:51.137081 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.137055 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.168850 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.168819 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw"] Oct 30 01:49:51.219657 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.219621 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:51.221160 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.221140 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-78d5ff\"/\"premium-w3-78d5ff-kserve-self-signed-certs\"" Oct 30 01:49:51.237587 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.237556 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-home\") pod \"premium-w3-78d5ff-kserve-7b944bddc9-8fnfw\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.237762 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.237608 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63663089-b7d3-4565-a95a-3b06e913271b-tls-certs\") pod \"premium-w3-78d5ff-kserve-7b944bddc9-8fnfw\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.237839 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.237796 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m68z7\" (UniqueName: \"kubernetes.io/projected/63663089-b7d3-4565-a95a-3b06e913271b-kube-api-access-m68z7\") pod \"premium-w3-78d5ff-kserve-7b944bddc9-8fnfw\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.237902 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.237841 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-tmp-dir\") pod \"premium-w3-78d5ff-kserve-7b944bddc9-8fnfw\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.237902 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.237858 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-model-cache\") pod \"premium-w3-78d5ff-kserve-7b944bddc9-8fnfw\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.238039 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.238015 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-dshm\") pod \"premium-w3-78d5ff-kserve-7b944bddc9-8fnfw\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.339172 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.339117 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m68z7\" (UniqueName: \"kubernetes.io/projected/63663089-b7d3-4565-a95a-3b06e913271b-kube-api-access-m68z7\") pod \"premium-w3-78d5ff-kserve-7b944bddc9-8fnfw\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.339381 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.339203 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-tmp-dir\") pod \"premium-w3-78d5ff-kserve-7b944bddc9-8fnfw\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.339381 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.339234 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-model-cache\") pod \"premium-w3-78d5ff-kserve-7b944bddc9-8fnfw\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.339381 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.339280 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-dshm\") pod \"premium-w3-78d5ff-kserve-7b944bddc9-8fnfw\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.339381 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.339353 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-home\") pod \"premium-w3-78d5ff-kserve-7b944bddc9-8fnfw\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.339609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.339400 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63663089-b7d3-4565-a95a-3b06e913271b-tls-certs\") pod \"premium-w3-78d5ff-kserve-7b944bddc9-8fnfw\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.340604 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.340401 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-tmp-dir\") pod \"premium-w3-78d5ff-kserve-7b944bddc9-8fnfw\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.340741 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.340703 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-home\") pod \"premium-w3-78d5ff-kserve-7b944bddc9-8fnfw\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.341014 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.340993 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-model-cache\") pod \"premium-w3-78d5ff-kserve-7b944bddc9-8fnfw\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.344657 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.344629 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63663089-b7d3-4565-a95a-3b06e913271b-tls-certs\") pod \"premium-w3-78d5ff-kserve-7b944bddc9-8fnfw\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.347395 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.347361 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-dshm\") pod \"premium-w3-78d5ff-kserve-7b944bddc9-8fnfw\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.365426 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.365394 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m68z7\" (UniqueName: \"kubernetes.io/projected/63663089-b7d3-4565-a95a-3b06e913271b-kube-api-access-m68z7\") pod \"premium-w3-78d5ff-kserve-7b944bddc9-8fnfw\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.451451 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.451418 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:51.667572 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.667520 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:51.750769 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:51.750672 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw"] Oct 30 01:49:51.755467 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:49:51.755436 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63663089_b7d3_4565_a95a_3b06e913271b.slice/crio-0de8f9dbfd374e39c41e62fd43e1aea599a8518b9335bf413bdd3b7b2cce8bdd WatchSource:0}: Error finding container 0de8f9dbfd374e39c41e62fd43e1aea599a8518b9335bf413bdd3b7b2cce8bdd: Status 404 returned error can't find the container with id 0de8f9dbfd374e39c41e62fd43e1aea599a8518b9335bf413bdd3b7b2cce8bdd Oct 30 01:49:52.690516 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:52.690479 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" event={"ID":"63663089-b7d3-4565-a95a-3b06e913271b","Type":"ContainerStarted","Data":"c093e6c10ab0574dc2a9a8192a1a6a199ca755800651847952260f2896895ade"} Oct 30 01:49:52.690929 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:52.690524 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" event={"ID":"63663089-b7d3-4565-a95a-3b06e913271b","Type":"ContainerStarted","Data":"0de8f9dbfd374e39c41e62fd43e1aea599a8518b9335bf413bdd3b7b2cce8bdd"} Oct 30 01:49:52.731824 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:52.731764 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podStartSLOduration=1.731744859 podStartE2EDuration="1.731744859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:52.731206319 +0000 UTC m=+1222.309997413" watchObservedRunningTime="2026-10-30 01:49:52.731744859 +0000 UTC m=+1222.310535934" Oct 30 01:49:53.279177 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:53.279142 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:53.452149 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:53.452081 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:49:53.907766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:53.907724 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:49:54.892035 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:54.891995 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:55.221386 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:49:55.221340 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:49:56.483740 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:56.483699 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:56.606419 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:56.606380 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:58.068165 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:58.068132 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:49:59.565662 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:49:59.565610 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:01.356128 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:01.356073 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:01.451657 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:01.451618 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:50:01.465031 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:01.465003 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:50:01.765653 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:01.765621 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:50:03.135700 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:03.135660 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:04.728142 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:04.728082 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:06.212338 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.212291 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:06.339771 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.339723 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv"] Oct 30 01:50:06.345891 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.345862 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.375823 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.375794 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f6f4f0\"/\"kube-root-ca.crt\"" Oct 30 01:50:06.375823 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.375794 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-f6f4f0\"/\"openshift-service-ca.crt\"" Oct 30 01:50:06.376019 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.375850 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f6f4f0\"/\"default-dockercfg-hv9nq\"" Oct 30 01:50:06.376019 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.375951 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f6f4f0\"/\"simulator-w1-f6f4f0-kserve-self-signed-certs\"" Oct 30 01:50:06.379070 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.379045 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv"] Oct 30 01:50:06.385627 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.385599 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfd9t\" (UniqueName: \"kubernetes.io/projected/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-kube-api-access-mfd9t\") pod \"simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.385762 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.385741 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-model-cache\") pod \"simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.385805 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.385781 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-tmp-dir\") pod \"simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.385865 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.385849 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-tls-certs\") pod \"simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.385946 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.385921 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-home\") pod \"simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.385997 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.385982 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-dshm\") pod \"simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.402278 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.402241 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:06.487565 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.487475 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-model-cache\") pod \"simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.487565 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.487517 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-tmp-dir\") pod \"simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.487565 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.487546 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-tls-certs\") pod \"simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.487833 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.487576 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-home\") pod \"simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.487833 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.487620 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-dshm\") pod \"simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.487833 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.487691 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfd9t\" (UniqueName: \"kubernetes.io/projected/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-kube-api-access-mfd9t\") pod \"simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.487999 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.487951 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-model-cache\") pod \"simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.488342 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.488311 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-home\") pod \"simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.488483 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.488342 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-tmp-dir\") pod \"simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.490149 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.490123 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-dshm\") pod \"simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.490443 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.490413 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-tls-certs\") pod \"simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.511002 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.510962 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfd9t\" (UniqueName: \"kubernetes.io/projected/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-kube-api-access-mfd9t\") pod \"simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.660221 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.660182 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:06.974188 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:06.973859 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv"] Oct 30 01:50:07.287152 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:07.287033 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:50:07.787667 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:07.787631 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" event={"ID":"448d43d9-9a90-4567-a9af-d5b9dd4c20ac","Type":"ContainerStarted","Data":"75d9abe139b45165f1b87576b669136870cbeb1931bd3c2ee65464aec131419a"} Oct 30 01:50:07.787853 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:07.787675 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" event={"ID":"448d43d9-9a90-4567-a9af-d5b9dd4c20ac","Type":"ContainerStarted","Data":"46419f88e3022f204dba9afb0cc8710e2458015b28265368f7a10d1d0df4c9fa"} Oct 30 01:50:07.830526 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:07.830473 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podStartSLOduration=1.830457296 podStartE2EDuration="1.830457296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:50:07.82922254 +0000 UTC m=+1237.408013621" watchObservedRunningTime="2026-10-30 01:50:07.830457296 +0000 UTC m=+1237.409248377" Oct 30 01:50:07.879076 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:07.879040 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:08.660815 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:08.660780 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:09.391234 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:09.391197 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:10.877496 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:10.877466 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:12.073766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:12.073703 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:12.402319 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:12.402236 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:14.188040 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:14.187977 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:15.779846 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:15.779806 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:16.661114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:16.661054 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:16.673823 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:16.673789 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:16.856447 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:16.856416 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:50:17.468706 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:17.468671 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:18.392378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:18.392342 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:50:19.197735 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:19.197692 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:20.771965 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:20.771933 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:22.374525 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:22.374490 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:23.974568 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:23.974534 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:24.405017 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:24.404934 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:25.216663 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:50:25.216628 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:50:25.686186 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:25.686156 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:27.291502 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:27.291462 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:28.891311 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:28.891270 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:30.366214 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:30.366172 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:50:30.594937 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:30.594887 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:31.112578 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.112536 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:31.216806 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.216758 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:31.638454 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.638414 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7"] Oct 30 01:50:31.643000 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.642974 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.669292 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.669260 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f6f4f0\"/\"premium-w1-f6f4f0-kserve-self-signed-certs\"" Oct 30 01:50:31.671500 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.671472 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7"] Oct 30 01:50:31.700495 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.700424 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:31.744535 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.744498 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-model-cache\") pod \"premium-w1-f6f4f0-kserve-674f58d54f-gqpq7\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.744714 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.744620 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-dshm\") pod \"premium-w1-f6f4f0-kserve-674f58d54f-gqpq7\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.744714 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.744694 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/161dcfca-cef5-45b8-983d-3bffa46cc52d-tls-certs\") pod \"premium-w1-f6f4f0-kserve-674f58d54f-gqpq7\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.744803 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.744745 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-home\") pod \"premium-w1-f6f4f0-kserve-674f58d54f-gqpq7\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.744865 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.744847 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-tmp-dir\") pod \"premium-w1-f6f4f0-kserve-674f58d54f-gqpq7\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.744902 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.744884 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6h2r\" (UniqueName: \"kubernetes.io/projected/161dcfca-cef5-45b8-983d-3bffa46cc52d-kube-api-access-c6h2r\") pod \"premium-w1-f6f4f0-kserve-674f58d54f-gqpq7\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.846301 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.846266 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-model-cache\") pod \"premium-w1-f6f4f0-kserve-674f58d54f-gqpq7\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.846519 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.846319 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-dshm\") pod \"premium-w1-f6f4f0-kserve-674f58d54f-gqpq7\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.846519 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.846343 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/161dcfca-cef5-45b8-983d-3bffa46cc52d-tls-certs\") pod \"premium-w1-f6f4f0-kserve-674f58d54f-gqpq7\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.846519 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.846381 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-home\") pod \"premium-w1-f6f4f0-kserve-674f58d54f-gqpq7\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.846519 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.846419 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-tmp-dir\") pod \"premium-w1-f6f4f0-kserve-674f58d54f-gqpq7\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.846519 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.846466 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6h2r\" (UniqueName: \"kubernetes.io/projected/161dcfca-cef5-45b8-983d-3bffa46cc52d-kube-api-access-c6h2r\") pod \"premium-w1-f6f4f0-kserve-674f58d54f-gqpq7\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.847187 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.847161 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-model-cache\") pod \"premium-w1-f6f4f0-kserve-674f58d54f-gqpq7\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.847708 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.847681 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-home\") pod \"premium-w1-f6f4f0-kserve-674f58d54f-gqpq7\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.847995 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.847972 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-tmp-dir\") pod \"premium-w1-f6f4f0-kserve-674f58d54f-gqpq7\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.851401 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.851353 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-dshm\") pod \"premium-w1-f6f4f0-kserve-674f58d54f-gqpq7\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.851507 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.851427 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/161dcfca-cef5-45b8-983d-3bffa46cc52d-tls-certs\") pod \"premium-w1-f6f4f0-kserve-674f58d54f-gqpq7\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.869299 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.869270 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6h2r\" (UniqueName: \"kubernetes.io/projected/161dcfca-cef5-45b8-983d-3bffa46cc52d-kube-api-access-c6h2r\") pod \"premium-w1-f6f4f0-kserve-674f58d54f-gqpq7\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:31.957285 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:31.957245 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:32.260846 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.260441 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7"] Oct 30 01:50:32.265574 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:50:32.265545 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod161dcfca_cef5_45b8_983d_3bffa46cc52d.slice/crio-9031be1ac4d0f3e881bca2bdfa8607875c2f4a7a8de29cae6e58b890d145303c WatchSource:0}: Error finding container 9031be1ac4d0f3e881bca2bdfa8607875c2f4a7a8de29cae6e58b890d145303c: Status 404 returned error can't find the container with id 9031be1ac4d0f3e881bca2bdfa8607875c2f4a7a8de29cae6e58b890d145303c Oct 30 01:50:32.277028 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.276984 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:32.308145 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.308116 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k"] Oct 30 01:50:32.313812 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.313792 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k" Oct 30 01:50:32.330450 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.330411 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:50:32.330658 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.330640 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:50:32.330743 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.330673 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:50:32.330743 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.330707 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:50:32.335071 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.334263 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k"] Oct 30 01:50:32.453199 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.453165 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pwlq\" (UniqueName: \"kubernetes.io/projected/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-kube-api-access-8pwlq\") pod \"maas-api-revoke-sub-80ca7ecc-p9c2k\" (UID: \"7dbdeeef-b6cd-43a2-b969-458bb5cbc302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k" Oct 30 01:50:32.453403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.453237 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80ca7ecc-p9c2k\" (UID: \"7dbdeeef-b6cd-43a2-b969-458bb5cbc302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k" Oct 30 01:50:32.453403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.453357 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80ca7ecc-p9c2k\" (UID: \"7dbdeeef-b6cd-43a2-b969-458bb5cbc302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k" Oct 30 01:50:32.554116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.554022 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80ca7ecc-p9c2k\" (UID: \"7dbdeeef-b6cd-43a2-b969-458bb5cbc302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k" Oct 30 01:50:32.554272 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.554119 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pwlq\" (UniqueName: \"kubernetes.io/projected/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-kube-api-access-8pwlq\") pod \"maas-api-revoke-sub-80ca7ecc-p9c2k\" (UID: \"7dbdeeef-b6cd-43a2-b969-458bb5cbc302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k" Oct 30 01:50:32.554272 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.554158 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80ca7ecc-p9c2k\" (UID: \"7dbdeeef-b6cd-43a2-b969-458bb5cbc302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k" Oct 30 01:50:32.554767 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.554747 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80ca7ecc-p9c2k\" (UID: \"7dbdeeef-b6cd-43a2-b969-458bb5cbc302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k" Oct 30 01:50:32.556418 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.556401 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80ca7ecc-p9c2k\" (UID: \"7dbdeeef-b6cd-43a2-b969-458bb5cbc302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k" Oct 30 01:50:32.576705 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.576673 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pwlq\" (UniqueName: \"kubernetes.io/projected/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-kube-api-access-8pwlq\") pod \"maas-api-revoke-sub-80ca7ecc-p9c2k\" (UID: \"7dbdeeef-b6cd-43a2-b969-458bb5cbc302\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k" Oct 30 01:50:32.632083 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.632047 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k" Oct 30 01:50:32.934374 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.934345 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k"] Oct 30 01:50:32.935675 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.935642 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" event={"ID":"161dcfca-cef5-45b8-983d-3bffa46cc52d","Type":"ContainerStarted","Data":"a2b68025459301444789f6d51fa02f0936be8351d5f087b43a269cbbf8473a27"} Oct 30 01:50:32.935804 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.935691 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" event={"ID":"161dcfca-cef5-45b8-983d-3bffa46cc52d","Type":"ContainerStarted","Data":"9031be1ac4d0f3e881bca2bdfa8607875c2f4a7a8de29cae6e58b890d145303c"} Oct 30 01:50:32.939377 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:50:32.939352 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbdeeef_b6cd_43a2_b969_458bb5cbc302.slice/crio-2eed578b54831794af0b471e41fb1daf048d99b414eded6281947fd3348c8c77 WatchSource:0}: Error finding container 2eed578b54831794af0b471e41fb1daf048d99b414eded6281947fd3348c8c77: Status 404 returned error can't find the container with id 2eed578b54831794af0b471e41fb1daf048d99b414eded6281947fd3348c8c77 Oct 30 01:50:32.957407 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.957385 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:32.977518 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:32.977454 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podStartSLOduration=1.977439585 podStartE2EDuration="1.977439585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:50:32.975386088 +0000 UTC m=+1262.554177168" watchObservedRunningTime="2026-10-30 01:50:32.977439585 +0000 UTC m=+1262.556230665" Oct 30 01:50:33.920303 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:33.920263 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:33.942777 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:33.942741 5998 generic.go:356] "Generic (PLEG): container finished" podID="7dbdeeef-b6cd-43a2-b969-458bb5cbc302" containerID="462b045160cd3daa9c3777af2725fe95488ab3d61990ef24ef48963a04feab2d" exitCode=0 Oct 30 01:50:33.943221 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:33.942837 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k" event={"ID":"7dbdeeef-b6cd-43a2-b969-458bb5cbc302","Type":"ContainerDied","Data":"462b045160cd3daa9c3777af2725fe95488ab3d61990ef24ef48963a04feab2d"} Oct 30 01:50:33.943221 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:33.942890 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k" event={"ID":"7dbdeeef-b6cd-43a2-b969-458bb5cbc302","Type":"ContainerStarted","Data":"2eed578b54831794af0b471e41fb1daf048d99b414eded6281947fd3348c8c77"} Oct 30 01:50:34.029880 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:34.029848 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:35.125893 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:35.125866 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k" Oct 30 01:50:35.299238 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:35.299152 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-maas-api-openshift-ca\") pod \"7dbdeeef-b6cd-43a2-b969-458bb5cbc302\" (UID: \"7dbdeeef-b6cd-43a2-b969-458bb5cbc302\") " Oct 30 01:50:35.299416 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:35.299300 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-kube-api-access-8pwlq\" (UniqueName: \"kubernetes.io/projected/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-kube-api-access-8pwlq\") pod \"7dbdeeef-b6cd-43a2-b969-458bb5cbc302\" (UID: \"7dbdeeef-b6cd-43a2-b969-458bb5cbc302\") " Oct 30 01:50:35.299416 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:35.299341 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-maas-api-xks-ca\") pod \"7dbdeeef-b6cd-43a2-b969-458bb5cbc302\" (UID: \"7dbdeeef-b6cd-43a2-b969-458bb5cbc302\") " Oct 30 01:50:35.299541 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:35.299515 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-maas-api-openshift-ca" pod "7dbdeeef-b6cd-43a2-b969-458bb5cbc302" (UID: "7dbdeeef-b6cd-43a2-b969-458bb5cbc302"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:50:35.299697 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:35.299681 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:50:35.301547 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:35.301513 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-kube-api-access-8pwlq" pod "7dbdeeef-b6cd-43a2-b969-458bb5cbc302" (UID: "7dbdeeef-b6cd-43a2-b969-458bb5cbc302"). InnerVolumeSpecName "kube-api-access-8pwlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:50:35.301547 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:35.301525 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-maas-api-xks-ca" pod "7dbdeeef-b6cd-43a2-b969-458bb5cbc302" (UID: "7dbdeeef-b6cd-43a2-b969-458bb5cbc302"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:50:35.400797 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:35.400759 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pwlq\" (UniqueName: \"kubernetes.io/projected/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-kube-api-access-8pwlq\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:50:35.400797 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:35.400793 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7dbdeeef-b6cd-43a2-b969-458bb5cbc302-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:50:35.491195 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:35.491161 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:35.955329 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:35.955294 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k" Oct 30 01:50:35.955329 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:35.955307 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k" event={"ID":"7dbdeeef-b6cd-43a2-b969-458bb5cbc302","Type":"ContainerDied","Data":"2eed578b54831794af0b471e41fb1daf048d99b414eded6281947fd3348c8c77"} Oct 30 01:50:35.955553 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:35.955363 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eed578b54831794af0b471e41fb1daf048d99b414eded6281947fd3348c8c77" Oct 30 01:50:37.066905 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:37.066870 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:38.792292 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:38.792259 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:40.479444 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:40.479406 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:41.957507 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:41.957422 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:41.970889 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:41.970856 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:41.976270 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:41.976246 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:41.999173 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:41.999145 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:50:43.585441 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:43.585407 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:44.908025 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:44.907988 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r"] Oct 30 01:50:44.908735 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:44.908714 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7dbdeeef-b6cd-43a2-b969-458bb5cbc302" containerName="revoke-keys" Oct 30 01:50:44.908856 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:44.908737 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dbdeeef-b6cd-43a2-b969-458bb5cbc302" containerName="revoke-keys" Oct 30 01:50:44.908914 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:44.908878 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dbdeeef-b6cd-43a2-b969-458bb5cbc302" containerName="revoke-keys" Oct 30 01:50:44.913235 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:44.913211 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r" Oct 30 01:50:44.942147 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:44.942121 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:50:44.942382 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:44.942176 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:50:44.942450 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:44.942336 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:50:44.942450 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:44.942340 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:50:44.944693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:44.944669 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r"] Oct 30 01:50:44.988962 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:44.988928 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3013c307-263e-49ae-b79d-573925f7e85c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b07babfb-9z79r\" (UID: \"3013c307-263e-49ae-b79d-573925f7e85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r" Oct 30 01:50:44.989229 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:44.988982 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grjrd\" (UniqueName: \"kubernetes.io/projected/3013c307-263e-49ae-b79d-573925f7e85c-kube-api-access-grjrd\") pod \"maas-api-revoke-sub-b07babfb-9z79r\" (UID: \"3013c307-263e-49ae-b79d-573925f7e85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r" Oct 30 01:50:44.989229 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:44.989072 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3013c307-263e-49ae-b79d-573925f7e85c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b07babfb-9z79r\" (UID: \"3013c307-263e-49ae-b79d-573925f7e85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r" Oct 30 01:50:45.083636 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:45.083599 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:45.090369 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:45.090332 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3013c307-263e-49ae-b79d-573925f7e85c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b07babfb-9z79r\" (UID: \"3013c307-263e-49ae-b79d-573925f7e85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r" Oct 30 01:50:45.090524 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:45.090413 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grjrd\" (UniqueName: \"kubernetes.io/projected/3013c307-263e-49ae-b79d-573925f7e85c-kube-api-access-grjrd\") pod \"maas-api-revoke-sub-b07babfb-9z79r\" (UID: \"3013c307-263e-49ae-b79d-573925f7e85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r" Oct 30 01:50:45.090524 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:45.090463 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3013c307-263e-49ae-b79d-573925f7e85c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b07babfb-9z79r\" (UID: \"3013c307-263e-49ae-b79d-573925f7e85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r" Oct 30 01:50:45.091027 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:45.091003 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3013c307-263e-49ae-b79d-573925f7e85c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b07babfb-9z79r\" (UID: \"3013c307-263e-49ae-b79d-573925f7e85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r" Oct 30 01:50:45.092785 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:45.092763 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3013c307-263e-49ae-b79d-573925f7e85c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b07babfb-9z79r\" (UID: \"3013c307-263e-49ae-b79d-573925f7e85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r" Oct 30 01:50:45.113382 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:45.113350 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grjrd\" (UniqueName: \"kubernetes.io/projected/3013c307-263e-49ae-b79d-573925f7e85c-kube-api-access-grjrd\") pod \"maas-api-revoke-sub-b07babfb-9z79r\" (UID: \"3013c307-263e-49ae-b79d-573925f7e85c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r" Oct 30 01:50:45.228965 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:45.228914 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r" Oct 30 01:50:45.537881 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:45.537752 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r"] Oct 30 01:50:45.543301 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:50:45.543269 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3013c307_263e_49ae_b79d_573925f7e85c.slice/crio-d31cb92cac224830349600f50e65e1d0a59a03adfc9d1abbe8876b951d315686 WatchSource:0}: Error finding container d31cb92cac224830349600f50e65e1d0a59a03adfc9d1abbe8876b951d315686: Status 404 returned error can't find the container with id d31cb92cac224830349600f50e65e1d0a59a03adfc9d1abbe8876b951d315686 Oct 30 01:50:46.014149 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:46.014106 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r" event={"ID":"3013c307-263e-49ae-b79d-573925f7e85c","Type":"ContainerStarted","Data":"95bf7af10a3f8e92db83702c320a12e4da4997ada112d76d67e679e303ff27ba"} Oct 30 01:50:46.014636 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:46.014162 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r" event={"ID":"3013c307-263e-49ae-b79d-573925f7e85c","Type":"ContainerStarted","Data":"d31cb92cac224830349600f50e65e1d0a59a03adfc9d1abbe8876b951d315686"} Oct 30 01:50:46.054525 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:46.054473 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r" podStartSLOduration=2.054458681 podStartE2EDuration="2.054458681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:50:46.053496456 +0000 UTC m=+1275.632287538" watchObservedRunningTime="2026-10-30 01:50:46.054458681 +0000 UTC m=+1275.633249768" Oct 30 01:50:46.680334 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:46.680299 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:47.022537 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:47.022451 5998 generic.go:356] "Generic (PLEG): container finished" podID="3013c307-263e-49ae-b79d-573925f7e85c" containerID="95bf7af10a3f8e92db83702c320a12e4da4997ada112d76d67e679e303ff27ba" exitCode=0 Oct 30 01:50:47.022537 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:47.022505 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r" event={"ID":"3013c307-263e-49ae-b79d-573925f7e85c","Type":"ContainerDied","Data":"95bf7af10a3f8e92db83702c320a12e4da4997ada112d76d67e679e303ff27ba"} Oct 30 01:50:48.163442 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:48.163419 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r" Oct 30 01:50:48.246289 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:48.245820 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3013c307-263e-49ae-b79d-573925f7e85c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3013c307-263e-49ae-b79d-573925f7e85c-maas-api-openshift-ca\") pod \"3013c307-263e-49ae-b79d-573925f7e85c\" (UID: \"3013c307-263e-49ae-b79d-573925f7e85c\") " Oct 30 01:50:48.246289 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:48.245950 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3013c307-263e-49ae-b79d-573925f7e85c-kube-api-access-grjrd\" (UniqueName: \"kubernetes.io/projected/3013c307-263e-49ae-b79d-573925f7e85c-kube-api-access-grjrd\") pod \"3013c307-263e-49ae-b79d-573925f7e85c\" (UID: \"3013c307-263e-49ae-b79d-573925f7e85c\") " Oct 30 01:50:48.246289 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:48.246067 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3013c307-263e-49ae-b79d-573925f7e85c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3013c307-263e-49ae-b79d-573925f7e85c-maas-api-xks-ca\") pod \"3013c307-263e-49ae-b79d-573925f7e85c\" (UID: \"3013c307-263e-49ae-b79d-573925f7e85c\") " Oct 30 01:50:48.246289 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:48.246248 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3013c307-263e-49ae-b79d-573925f7e85c-maas-api-openshift-ca" pod "3013c307-263e-49ae-b79d-573925f7e85c" (UID: "3013c307-263e-49ae-b79d-573925f7e85c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:50:48.246655 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:48.246425 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3013c307-263e-49ae-b79d-573925f7e85c-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:50:48.248782 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:48.248752 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3013c307-263e-49ae-b79d-573925f7e85c-kube-api-access-grjrd" pod "3013c307-263e-49ae-b79d-573925f7e85c" (UID: "3013c307-263e-49ae-b79d-573925f7e85c"). InnerVolumeSpecName "kube-api-access-grjrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:50:48.248782 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:48.248760 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3013c307-263e-49ae-b79d-573925f7e85c-maas-api-xks-ca" pod "3013c307-263e-49ae-b79d-573925f7e85c" (UID: "3013c307-263e-49ae-b79d-573925f7e85c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:50:48.290384 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:48.290299 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:48.348000 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:48.347955 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3013c307-263e-49ae-b79d-573925f7e85c-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:50:48.348000 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:48.347997 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grjrd\" (UniqueName: \"kubernetes.io/projected/3013c307-263e-49ae-b79d-573925f7e85c-kube-api-access-grjrd\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:50:49.035988 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:49.035951 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r" Oct 30 01:50:49.035988 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:49.035974 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r" event={"ID":"3013c307-263e-49ae-b79d-573925f7e85c","Type":"ContainerDied","Data":"d31cb92cac224830349600f50e65e1d0a59a03adfc9d1abbe8876b951d315686"} Oct 30 01:50:49.036222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:49.036029 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d31cb92cac224830349600f50e65e1d0a59a03adfc9d1abbe8876b951d315686" Oct 30 01:50:49.222266 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:49.222234 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:50:49.969193 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:49.969157 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:51.535291 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:51.535245 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:53.130633 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:53.130593 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:54.614106 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:54.614058 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:55.216079 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:50:55.216042 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:50:56.249688 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:56.249650 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:57.889218 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:57.889185 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:59.196696 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.196659 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk"] Oct 30 01:50:59.197571 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.197541 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3013c307-263e-49ae-b79d-573925f7e85c" containerName="revoke-keys" Oct 30 01:50:59.197571 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.197565 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3013c307-263e-49ae-b79d-573925f7e85c" containerName="revoke-keys" Oct 30 01:50:59.197728 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.197625 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3013c307-263e-49ae-b79d-573925f7e85c" containerName="revoke-keys" Oct 30 01:50:59.203189 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.203160 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk" Oct 30 01:50:59.219105 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.219067 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:50:59.230665 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.230636 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:50:59.230809 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.230716 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:50:59.230875 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.230829 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:50:59.233574 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.233551 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk"] Oct 30 01:50:59.360204 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.360164 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5pmf\" (UniqueName: \"kubernetes.io/projected/d520569f-2b5b-4bb4-862e-6042401aadf8-kube-api-access-c5pmf\") pod \"maas-api-revoke-sub-d22ba8d6-cswkk\" (UID: \"d520569f-2b5b-4bb4-862e-6042401aadf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk" Oct 30 01:50:59.360387 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.360290 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d520569f-2b5b-4bb4-862e-6042401aadf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d22ba8d6-cswkk\" (UID: \"d520569f-2b5b-4bb4-862e-6042401aadf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk" Oct 30 01:50:59.360387 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.360368 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d520569f-2b5b-4bb4-862e-6042401aadf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d22ba8d6-cswkk\" (UID: \"d520569f-2b5b-4bb4-862e-6042401aadf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk" Oct 30 01:50:59.461628 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.461528 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d520569f-2b5b-4bb4-862e-6042401aadf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d22ba8d6-cswkk\" (UID: \"d520569f-2b5b-4bb4-862e-6042401aadf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk" Oct 30 01:50:59.461628 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.461620 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d520569f-2b5b-4bb4-862e-6042401aadf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d22ba8d6-cswkk\" (UID: \"d520569f-2b5b-4bb4-862e-6042401aadf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk" Oct 30 01:50:59.461856 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.461688 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5pmf\" (UniqueName: \"kubernetes.io/projected/d520569f-2b5b-4bb4-862e-6042401aadf8-kube-api-access-c5pmf\") pod \"maas-api-revoke-sub-d22ba8d6-cswkk\" (UID: \"d520569f-2b5b-4bb4-862e-6042401aadf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk" Oct 30 01:50:59.462381 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.462355 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d520569f-2b5b-4bb4-862e-6042401aadf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d22ba8d6-cswkk\" (UID: \"d520569f-2b5b-4bb4-862e-6042401aadf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk" Oct 30 01:50:59.463977 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.463947 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d520569f-2b5b-4bb4-862e-6042401aadf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d22ba8d6-cswkk\" (UID: \"d520569f-2b5b-4bb4-862e-6042401aadf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk" Oct 30 01:50:59.484050 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.484021 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5pmf\" (UniqueName: \"kubernetes.io/projected/d520569f-2b5b-4bb4-862e-6042401aadf8-kube-api-access-c5pmf\") pod \"maas-api-revoke-sub-d22ba8d6-cswkk\" (UID: \"d520569f-2b5b-4bb4-862e-6042401aadf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk" Oct 30 01:50:59.517983 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.517951 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk" Oct 30 01:50:59.584732 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.584662 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:50:59.820757 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:50:59.820675 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk"] Oct 30 01:50:59.824916 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:50:59.824886 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd520569f_2b5b_4bb4_862e_6042401aadf8.slice/crio-89aed7fcac4265da143f9a176e38effaf2004b5eb0dbcc618e5fdfac64f07a74 WatchSource:0}: Error finding container 89aed7fcac4265da143f9a176e38effaf2004b5eb0dbcc618e5fdfac64f07a74: Status 404 returned error can't find the container with id 89aed7fcac4265da143f9a176e38effaf2004b5eb0dbcc618e5fdfac64f07a74 Oct 30 01:51:00.100485 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:00.100393 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk" event={"ID":"d520569f-2b5b-4bb4-862e-6042401aadf8","Type":"ContainerStarted","Data":"89aed7fcac4265da143f9a176e38effaf2004b5eb0dbcc618e5fdfac64f07a74"} Oct 30 01:51:01.107641 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:01.107601 5998 generic.go:356] "Generic (PLEG): container finished" podID="d520569f-2b5b-4bb4-862e-6042401aadf8" containerID="c1203f05c2aa1ed83b8cc1daddcec0cecb6bd218e12b99d0160229fa6237697f" exitCode=0 Oct 30 01:51:01.108052 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:01.107683 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk" event={"ID":"d520569f-2b5b-4bb4-862e-6042401aadf8","Type":"ContainerDied","Data":"c1203f05c2aa1ed83b8cc1daddcec0cecb6bd218e12b99d0160229fa6237697f"} Oct 30 01:51:01.293282 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:01.293243 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:02.256602 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:02.256577 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk" Oct 30 01:51:02.391157 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:02.391063 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d520569f-2b5b-4bb4-862e-6042401aadf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d520569f-2b5b-4bb4-862e-6042401aadf8-maas-api-xks-ca\") pod \"d520569f-2b5b-4bb4-862e-6042401aadf8\" (UID: \"d520569f-2b5b-4bb4-862e-6042401aadf8\") " Oct 30 01:51:02.391157 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:02.391138 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d520569f-2b5b-4bb4-862e-6042401aadf8-kube-api-access-c5pmf\" (UniqueName: \"kubernetes.io/projected/d520569f-2b5b-4bb4-862e-6042401aadf8-kube-api-access-c5pmf\") pod \"d520569f-2b5b-4bb4-862e-6042401aadf8\" (UID: \"d520569f-2b5b-4bb4-862e-6042401aadf8\") " Oct 30 01:51:02.391348 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:02.391171 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d520569f-2b5b-4bb4-862e-6042401aadf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d520569f-2b5b-4bb4-862e-6042401aadf8-maas-api-openshift-ca\") pod \"d520569f-2b5b-4bb4-862e-6042401aadf8\" (UID: \"d520569f-2b5b-4bb4-862e-6042401aadf8\") " Oct 30 01:51:02.391600 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:02.391576 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d520569f-2b5b-4bb4-862e-6042401aadf8-maas-api-openshift-ca" pod "d520569f-2b5b-4bb4-862e-6042401aadf8" (UID: "d520569f-2b5b-4bb4-862e-6042401aadf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:51:02.393309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:02.393280 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d520569f-2b5b-4bb4-862e-6042401aadf8-maas-api-xks-ca" pod "d520569f-2b5b-4bb4-862e-6042401aadf8" (UID: "d520569f-2b5b-4bb4-862e-6042401aadf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:51:02.393419 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:02.393400 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d520569f-2b5b-4bb4-862e-6042401aadf8-kube-api-access-c5pmf" pod "d520569f-2b5b-4bb4-862e-6042401aadf8" (UID: "d520569f-2b5b-4bb4-862e-6042401aadf8"). InnerVolumeSpecName "kube-api-access-c5pmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:51:02.492973 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:02.492937 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d520569f-2b5b-4bb4-862e-6042401aadf8-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:51:02.492973 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:02.492968 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5pmf\" (UniqueName: \"kubernetes.io/projected/d520569f-2b5b-4bb4-862e-6042401aadf8-kube-api-access-c5pmf\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:51:02.492973 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:02.492979 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d520569f-2b5b-4bb4-862e-6042401aadf8-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:51:02.976277 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:02.976244 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:03.122842 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:03.122811 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk" Oct 30 01:51:03.122842 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:03.122833 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk" event={"ID":"d520569f-2b5b-4bb4-862e-6042401aadf8","Type":"ContainerDied","Data":"89aed7fcac4265da143f9a176e38effaf2004b5eb0dbcc618e5fdfac64f07a74"} Oct 30 01:51:03.123058 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:03.122884 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89aed7fcac4265da143f9a176e38effaf2004b5eb0dbcc618e5fdfac64f07a74" Oct 30 01:51:04.577201 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:04.577163 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:06.082065 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:06.082032 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:07.772047 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:07.772009 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:09.490980 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:09.490946 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:09.803202 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:09.803105 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:51:11.316843 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:11.316806 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:13.017608 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:13.017570 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:14.817573 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:14.817529 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:16.725600 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:16.725554 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:17.733185 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.733151 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b"] Oct 30 01:51:17.733859 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.733837 5998 cpu_manager.go:414] "containerMap: removing container" podUID="d520569f-2b5b-4bb4-862e-6042401aadf8" containerName="revoke-keys" Oct 30 01:51:17.733947 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.733860 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d520569f-2b5b-4bb4-862e-6042401aadf8" containerName="revoke-keys" Oct 30 01:51:17.734004 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.733992 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="d520569f-2b5b-4bb4-862e-6042401aadf8" containerName="revoke-keys" Oct 30 01:51:17.738820 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.738788 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b" Oct 30 01:51:17.757194 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.756839 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b"] Oct 30 01:51:17.766155 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.766108 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:51:17.766155 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.766132 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:51:17.766983 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.766954 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:51:17.767155 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.767016 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:51:17.849206 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.848952 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d78788f8-0311-47f6-9126-afa411a940f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1683ded2-6v98b\" (UID: \"d78788f8-0311-47f6-9126-afa411a940f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b" Oct 30 01:51:17.849206 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.849034 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlf2t\" (UniqueName: \"kubernetes.io/projected/d78788f8-0311-47f6-9126-afa411a940f2-kube-api-access-vlf2t\") pod \"maas-api-revoke-sub-1683ded2-6v98b\" (UID: \"d78788f8-0311-47f6-9126-afa411a940f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b" Oct 30 01:51:17.849206 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.849075 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d78788f8-0311-47f6-9126-afa411a940f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1683ded2-6v98b\" (UID: \"d78788f8-0311-47f6-9126-afa411a940f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b" Oct 30 01:51:17.950627 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.950592 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d78788f8-0311-47f6-9126-afa411a940f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1683ded2-6v98b\" (UID: \"d78788f8-0311-47f6-9126-afa411a940f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b" Oct 30 01:51:17.950801 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.950650 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlf2t\" (UniqueName: \"kubernetes.io/projected/d78788f8-0311-47f6-9126-afa411a940f2-kube-api-access-vlf2t\") pod \"maas-api-revoke-sub-1683ded2-6v98b\" (UID: \"d78788f8-0311-47f6-9126-afa411a940f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b" Oct 30 01:51:17.950801 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.950688 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d78788f8-0311-47f6-9126-afa411a940f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1683ded2-6v98b\" (UID: \"d78788f8-0311-47f6-9126-afa411a940f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b" Oct 30 01:51:17.951508 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.951482 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d78788f8-0311-47f6-9126-afa411a940f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1683ded2-6v98b\" (UID: \"d78788f8-0311-47f6-9126-afa411a940f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b" Oct 30 01:51:17.953227 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.953193 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d78788f8-0311-47f6-9126-afa411a940f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1683ded2-6v98b\" (UID: \"d78788f8-0311-47f6-9126-afa411a940f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b" Oct 30 01:51:17.973452 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:17.973425 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlf2t\" (UniqueName: \"kubernetes.io/projected/d78788f8-0311-47f6-9126-afa411a940f2-kube-api-access-vlf2t\") pod \"maas-api-revoke-sub-1683ded2-6v98b\" (UID: \"d78788f8-0311-47f6-9126-afa411a940f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b" Oct 30 01:51:18.054484 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:18.054392 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b" Oct 30 01:51:18.366406 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:18.366330 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b"] Oct 30 01:51:18.370186 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:51:18.370156 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78788f8_0311_47f6_9126_afa411a940f2.slice/crio-59439a802ef00f29e73eeecb5ba3be60a9f206f8261aff0d5ec7f9f533857409 WatchSource:0}: Error finding container 59439a802ef00f29e73eeecb5ba3be60a9f206f8261aff0d5ec7f9f533857409: Status 404 returned error can't find the container with id 59439a802ef00f29e73eeecb5ba3be60a9f206f8261aff0d5ec7f9f533857409 Oct 30 01:51:18.439108 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:18.433482 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:18.879466 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:18.879429 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg"] Oct 30 01:51:18.884120 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:18.884082 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:18.910126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:18.910066 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-78d5ff\"/\"e2e-conflict-a-e338a073-kserve-self-signed-certs\"" Oct 30 01:51:18.913649 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:18.913613 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg"] Oct 30 01:51:18.935732 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:18.935699 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:18.963121 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:18.963006 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfmgq\" (UniqueName: \"kubernetes.io/projected/56e784e5-a048-4f8e-acec-0eeae021fcfb-kube-api-access-gfmgq\") pod \"e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:18.963121 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:18.963075 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-dshm\") pod \"e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:18.963340 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:18.963184 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-model-cache\") pod \"e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:18.963340 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:18.963238 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e784e5-a048-4f8e-acec-0eeae021fcfb-tls-certs\") pod \"e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:18.963448 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:18.963344 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-home\") pod \"e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:18.963448 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:18.963418 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-tmp-dir\") pod \"e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:19.064849 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:19.064808 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfmgq\" (UniqueName: \"kubernetes.io/projected/56e784e5-a048-4f8e-acec-0eeae021fcfb-kube-api-access-gfmgq\") pod \"e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:19.065040 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:19.064874 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-dshm\") pod \"e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:19.065040 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:19.064902 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-model-cache\") pod \"e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:19.065040 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:19.064961 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e784e5-a048-4f8e-acec-0eeae021fcfb-tls-certs\") pod \"e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:19.065040 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:19.064997 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-home\") pod \"e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:19.065301 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:19.065083 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-tmp-dir\") pod \"e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:19.065687 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:19.065656 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-home\") pod \"e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:19.065791 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:19.065694 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-model-cache\") pod \"e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:19.065851 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:19.065804 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-tmp-dir\") pod \"e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:19.068039 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:19.068013 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-dshm\") pod \"e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:19.068399 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:19.068370 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e784e5-a048-4f8e-acec-0eeae021fcfb-tls-certs\") pod \"e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:19.090791 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:19.090757 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfmgq\" (UniqueName: \"kubernetes.io/projected/56e784e5-a048-4f8e-acec-0eeae021fcfb-kube-api-access-gfmgq\") pod \"e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:19.199306 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:19.199264 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:19.219994 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:19.219927 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b" event={"ID":"d78788f8-0311-47f6-9126-afa411a940f2","Type":"ContainerStarted","Data":"d7e6c60cda2fc5088d5a126c182b5b4594bd5f20a1b4785c04066a3fedcd7458"} Oct 30 01:51:19.220136 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:19.220015 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b" event={"ID":"d78788f8-0311-47f6-9126-afa411a940f2","Type":"ContainerStarted","Data":"59439a802ef00f29e73eeecb5ba3be60a9f206f8261aff0d5ec7f9f533857409"} Oct 30 01:51:19.278296 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:19.278239 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b" podStartSLOduration=2.278224646 podStartE2EDuration="2.278224646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:19.276932171 +0000 UTC m=+1308.855723255" watchObservedRunningTime="2026-10-30 01:51:19.278224646 +0000 UTC m=+1308.857015726" Oct 30 01:51:19.504074 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:19.503983 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg"] Oct 30 01:51:19.508257 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:51:19.508214 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56e784e5_a048_4f8e_acec_0eeae021fcfb.slice/crio-0543969ce05c54f64b74522946d555b8dc491e8fb6c9d84d6345b439d5b4b15f WatchSource:0}: Error finding container 0543969ce05c54f64b74522946d555b8dc491e8fb6c9d84d6345b439d5b4b15f: Status 404 returned error can't find the container with id 0543969ce05c54f64b74522946d555b8dc491e8fb6c9d84d6345b439d5b4b15f Oct 30 01:51:20.162823 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:20.162789 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:20.227803 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:20.227689 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" event={"ID":"56e784e5-a048-4f8e-acec-0eeae021fcfb","Type":"ContainerStarted","Data":"75e072709ae61fe833f5fc2892320bac75235c6067b3be6cfd4cd20bdabc93ac"} Oct 30 01:51:20.227803 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:20.227739 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" event={"ID":"56e784e5-a048-4f8e-acec-0eeae021fcfb","Type":"ContainerStarted","Data":"0543969ce05c54f64b74522946d555b8dc491e8fb6c9d84d6345b439d5b4b15f"} Oct 30 01:51:20.229458 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:20.229428 5998 generic.go:356] "Generic (PLEG): container finished" podID="d78788f8-0311-47f6-9126-afa411a940f2" containerID="d7e6c60cda2fc5088d5a126c182b5b4594bd5f20a1b4785c04066a3fedcd7458" exitCode=0 Oct 30 01:51:20.229593 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:20.229515 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b" event={"ID":"d78788f8-0311-47f6-9126-afa411a940f2","Type":"ContainerDied","Data":"d7e6c60cda2fc5088d5a126c182b5b4594bd5f20a1b4785c04066a3fedcd7458"} Oct 30 01:51:20.271708 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:20.271641 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" podStartSLOduration=2.271619067 podStartE2EDuration="2.271619067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:20.269161104 +0000 UTC m=+1309.847952182" watchObservedRunningTime="2026-10-30 01:51:20.271619067 +0000 UTC m=+1309.850410149" Oct 30 01:51:21.199612 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:21.199556 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:21.383185 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:21.382877 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b" Oct 30 01:51:21.489118 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:21.488999 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d78788f8-0311-47f6-9126-afa411a940f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d78788f8-0311-47f6-9126-afa411a940f2-maas-api-xks-ca\") pod \"d78788f8-0311-47f6-9126-afa411a940f2\" (UID: \"d78788f8-0311-47f6-9126-afa411a940f2\") " Oct 30 01:51:21.489251 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:21.489142 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d78788f8-0311-47f6-9126-afa411a940f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d78788f8-0311-47f6-9126-afa411a940f2-maas-api-openshift-ca\") pod \"d78788f8-0311-47f6-9126-afa411a940f2\" (UID: \"d78788f8-0311-47f6-9126-afa411a940f2\") " Oct 30 01:51:21.489251 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:21.489243 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d78788f8-0311-47f6-9126-afa411a940f2-kube-api-access-vlf2t\" (UniqueName: \"kubernetes.io/projected/d78788f8-0311-47f6-9126-afa411a940f2-kube-api-access-vlf2t\") pod \"d78788f8-0311-47f6-9126-afa411a940f2\" (UID: \"d78788f8-0311-47f6-9126-afa411a940f2\") " Oct 30 01:51:21.489493 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:21.489472 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d78788f8-0311-47f6-9126-afa411a940f2-maas-api-openshift-ca" pod "d78788f8-0311-47f6-9126-afa411a940f2" (UID: "d78788f8-0311-47f6-9126-afa411a940f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:51:21.489664 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:21.489646 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d78788f8-0311-47f6-9126-afa411a940f2-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:51:21.491256 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:21.491236 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d78788f8-0311-47f6-9126-afa411a940f2-maas-api-xks-ca" pod "d78788f8-0311-47f6-9126-afa411a940f2" (UID: "d78788f8-0311-47f6-9126-afa411a940f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:51:21.491355 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:21.491341 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78788f8-0311-47f6-9126-afa411a940f2-kube-api-access-vlf2t" pod "d78788f8-0311-47f6-9126-afa411a940f2" (UID: "d78788f8-0311-47f6-9126-afa411a940f2"). InnerVolumeSpecName "kube-api-access-vlf2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:51:21.590632 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:21.590588 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlf2t\" (UniqueName: \"kubernetes.io/projected/d78788f8-0311-47f6-9126-afa411a940f2-kube-api-access-vlf2t\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:51:21.590632 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:21.590626 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d78788f8-0311-47f6-9126-afa411a940f2-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:51:21.674525 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:21.674483 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:22.245046 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:22.245016 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b" Oct 30 01:51:22.245556 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:22.245056 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b" event={"ID":"d78788f8-0311-47f6-9126-afa411a940f2","Type":"ContainerDied","Data":"59439a802ef00f29e73eeecb5ba3be60a9f206f8261aff0d5ec7f9f533857409"} Oct 30 01:51:22.245556 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:22.245133 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59439a802ef00f29e73eeecb5ba3be60a9f206f8261aff0d5ec7f9f533857409" Oct 30 01:51:23.273216 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:23.273178 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:24.867052 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:24.867010 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:24.983891 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:24.983855 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:51:25.219322 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:51:25.219283 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:51:26.761377 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:26.761342 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:28.051938 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.051840 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj"] Oct 30 01:51:28.052442 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.052287 5998 cpu_manager.go:414] "containerMap: removing container" podUID="d78788f8-0311-47f6-9126-afa411a940f2" containerName="revoke-keys" Oct 30 01:51:28.052442 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.052300 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d78788f8-0311-47f6-9126-afa411a940f2" containerName="revoke-keys" Oct 30 01:51:28.052442 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.052403 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="d78788f8-0311-47f6-9126-afa411a940f2" containerName="revoke-keys" Oct 30 01:51:28.055580 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.055561 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj" Oct 30 01:51:28.069951 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.069925 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:51:28.081783 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.081756 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:51:28.081922 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.081816 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:51:28.081922 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.081829 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:51:28.085344 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.085317 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj"] Oct 30 01:51:28.159048 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.159013 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d9393ac-db00-44c6-9fd3-8f767c77a796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a90a24a-hzxjj\" (UID: \"2d9393ac-db00-44c6-9fd3-8f767c77a796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj" Oct 30 01:51:28.159228 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.159178 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d9393ac-db00-44c6-9fd3-8f767c77a796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a90a24a-hzxjj\" (UID: \"2d9393ac-db00-44c6-9fd3-8f767c77a796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj" Oct 30 01:51:28.159228 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.159224 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq92w\" (UniqueName: \"kubernetes.io/projected/2d9393ac-db00-44c6-9fd3-8f767c77a796-kube-api-access-fq92w\") pod \"maas-api-revoke-sub-2a90a24a-hzxjj\" (UID: \"2d9393ac-db00-44c6-9fd3-8f767c77a796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj" Oct 30 01:51:28.260370 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.260311 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d9393ac-db00-44c6-9fd3-8f767c77a796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a90a24a-hzxjj\" (UID: \"2d9393ac-db00-44c6-9fd3-8f767c77a796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj" Oct 30 01:51:28.260555 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.260449 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d9393ac-db00-44c6-9fd3-8f767c77a796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a90a24a-hzxjj\" (UID: \"2d9393ac-db00-44c6-9fd3-8f767c77a796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj" Oct 30 01:51:28.260555 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.260507 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq92w\" (UniqueName: \"kubernetes.io/projected/2d9393ac-db00-44c6-9fd3-8f767c77a796-kube-api-access-fq92w\") pod \"maas-api-revoke-sub-2a90a24a-hzxjj\" (UID: \"2d9393ac-db00-44c6-9fd3-8f767c77a796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj" Oct 30 01:51:28.261293 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.261266 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d9393ac-db00-44c6-9fd3-8f767c77a796-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a90a24a-hzxjj\" (UID: \"2d9393ac-db00-44c6-9fd3-8f767c77a796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj" Oct 30 01:51:28.262781 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.262755 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d9393ac-db00-44c6-9fd3-8f767c77a796-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a90a24a-hzxjj\" (UID: \"2d9393ac-db00-44c6-9fd3-8f767c77a796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj" Oct 30 01:51:28.284128 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.284085 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq92w\" (UniqueName: \"kubernetes.io/projected/2d9393ac-db00-44c6-9fd3-8f767c77a796-kube-api-access-fq92w\") pod \"maas-api-revoke-sub-2a90a24a-hzxjj\" (UID: \"2d9393ac-db00-44c6-9fd3-8f767c77a796\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj" Oct 30 01:51:28.368878 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.368788 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj" Oct 30 01:51:28.391132 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.391103 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:28.675419 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:28.675382 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj"] Oct 30 01:51:28.680270 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:51:28.680238 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9393ac_db00_44c6_9fd3_8f767c77a796.slice/crio-7c3e232a5621dfccb0c925768dd4253dfcef5a897c24acc4d014ada4da7074ba WatchSource:0}: Error finding container 7c3e232a5621dfccb0c925768dd4253dfcef5a897c24acc4d014ada4da7074ba: Status 404 returned error can't find the container with id 7c3e232a5621dfccb0c925768dd4253dfcef5a897c24acc4d014ada4da7074ba Oct 30 01:51:29.199503 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:29.199459 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:29.213024 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:29.212993 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:29.286580 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:29.286489 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj" event={"ID":"2d9393ac-db00-44c6-9fd3-8f767c77a796","Type":"ContainerStarted","Data":"1a8bd0f9988c9ed2b1f6225055bb11c3758150417259b388b42ddbf57a7b1a8d"} Oct 30 01:51:29.286580 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:29.286552 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj" event={"ID":"2d9393ac-db00-44c6-9fd3-8f767c77a796","Type":"ContainerStarted","Data":"7c3e232a5621dfccb0c925768dd4253dfcef5a897c24acc4d014ada4da7074ba"} Oct 30 01:51:29.299132 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:29.299104 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:51:29.328506 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:29.328454 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj" podStartSLOduration=1.328436341 podStartE2EDuration="1.328436341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:29.328402749 +0000 UTC m=+1318.907193829" watchObservedRunningTime="2026-10-30 01:51:29.328436341 +0000 UTC m=+1318.907227421" Oct 30 01:51:30.063803 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:30.063767 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:30.295230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:30.295190 5998 generic.go:356] "Generic (PLEG): container finished" podID="2d9393ac-db00-44c6-9fd3-8f767c77a796" containerID="1a8bd0f9988c9ed2b1f6225055bb11c3758150417259b388b42ddbf57a7b1a8d" exitCode=0 Oct 30 01:51:30.295642 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:30.295265 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj" event={"ID":"2d9393ac-db00-44c6-9fd3-8f767c77a796","Type":"ContainerDied","Data":"1a8bd0f9988c9ed2b1f6225055bb11c3758150417259b388b42ddbf57a7b1a8d"} Oct 30 01:51:31.447646 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:31.447615 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj" Oct 30 01:51:31.597450 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:31.597362 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d9393ac-db00-44c6-9fd3-8f767c77a796-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d9393ac-db00-44c6-9fd3-8f767c77a796-maas-api-xks-ca\") pod \"2d9393ac-db00-44c6-9fd3-8f767c77a796\" (UID: \"2d9393ac-db00-44c6-9fd3-8f767c77a796\") " Oct 30 01:51:31.597450 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:31.597419 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d9393ac-db00-44c6-9fd3-8f767c77a796-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d9393ac-db00-44c6-9fd3-8f767c77a796-maas-api-openshift-ca\") pod \"2d9393ac-db00-44c6-9fd3-8f767c77a796\" (UID: \"2d9393ac-db00-44c6-9fd3-8f767c77a796\") " Oct 30 01:51:31.597645 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:31.597510 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d9393ac-db00-44c6-9fd3-8f767c77a796-kube-api-access-fq92w\" (UniqueName: \"kubernetes.io/projected/2d9393ac-db00-44c6-9fd3-8f767c77a796-kube-api-access-fq92w\") pod \"2d9393ac-db00-44c6-9fd3-8f767c77a796\" (UID: \"2d9393ac-db00-44c6-9fd3-8f767c77a796\") " Oct 30 01:51:31.597796 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:31.597768 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d9393ac-db00-44c6-9fd3-8f767c77a796-maas-api-openshift-ca" pod "2d9393ac-db00-44c6-9fd3-8f767c77a796" (UID: "2d9393ac-db00-44c6-9fd3-8f767c77a796"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:51:31.599538 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:31.599511 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d9393ac-db00-44c6-9fd3-8f767c77a796-maas-api-xks-ca" pod "2d9393ac-db00-44c6-9fd3-8f767c77a796" (UID: "2d9393ac-db00-44c6-9fd3-8f767c77a796"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:51:31.599668 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:31.599637 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9393ac-db00-44c6-9fd3-8f767c77a796-kube-api-access-fq92w" pod "2d9393ac-db00-44c6-9fd3-8f767c77a796" (UID: "2d9393ac-db00-44c6-9fd3-8f767c77a796"). InnerVolumeSpecName "kube-api-access-fq92w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:51:31.678209 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:31.678176 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:31.698659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:31.698627 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq92w\" (UniqueName: \"kubernetes.io/projected/2d9393ac-db00-44c6-9fd3-8f767c77a796-kube-api-access-fq92w\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:51:31.698659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:31.698658 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d9393ac-db00-44c6-9fd3-8f767c77a796-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:51:31.698830 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:31.698668 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d9393ac-db00-44c6-9fd3-8f767c77a796-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:51:32.318130 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:32.316884 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj" Oct 30 01:51:32.318130 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:32.316926 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj" event={"ID":"2d9393ac-db00-44c6-9fd3-8f767c77a796","Type":"ContainerDied","Data":"7c3e232a5621dfccb0c925768dd4253dfcef5a897c24acc4d014ada4da7074ba"} Oct 30 01:51:32.318130 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:32.316983 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c3e232a5621dfccb0c925768dd4253dfcef5a897c24acc4d014ada4da7074ba" Oct 30 01:51:33.192395 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:33.192358 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:34.082117 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:34.082066 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:34.191212 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:34.191181 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:34.797111 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:34.797040 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:36.364228 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.364187 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:36.488419 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.488385 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d"] Oct 30 01:51:36.488851 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.488836 5998 cpu_manager.go:414] "containerMap: removing container" podUID="2d9393ac-db00-44c6-9fd3-8f767c77a796" containerName="revoke-keys" Oct 30 01:51:36.488851 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.488851 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d9393ac-db00-44c6-9fd3-8f767c77a796" containerName="revoke-keys" Oct 30 01:51:36.488990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.488946 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d9393ac-db00-44c6-9fd3-8f767c77a796" containerName="revoke-keys" Oct 30 01:51:36.492581 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.492556 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.518222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.518191 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-f6f4f0\"/\"distinct-w1-f6f4f0-kserve-self-signed-certs\"" Oct 30 01:51:36.521218 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.521186 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d"] Oct 30 01:51:36.543938 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.543902 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:36.651106 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.651009 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-home\") pod \"distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.651284 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.651156 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-dshm\") pod \"distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.651284 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.651220 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-model-cache\") pod \"distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.651284 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.651268 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aafa0b2-1b3f-479a-89a5-2041cf887a21-tls-certs\") pod \"distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.651413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.651355 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2lvp\" (UniqueName: \"kubernetes.io/projected/3aafa0b2-1b3f-479a-89a5-2041cf887a21-kube-api-access-p2lvp\") pod \"distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.651413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.651398 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-tmp-dir\") pod \"distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.752362 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.752328 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-home\") pod \"distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.752548 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.752398 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-dshm\") pod \"distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.752548 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.752452 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-model-cache\") pod \"distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.752548 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.752491 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aafa0b2-1b3f-479a-89a5-2041cf887a21-tls-certs\") pod \"distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.752548 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.752522 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2lvp\" (UniqueName: \"kubernetes.io/projected/3aafa0b2-1b3f-479a-89a5-2041cf887a21-kube-api-access-p2lvp\") pod \"distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.752772 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.752563 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-tmp-dir\") pod \"distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.753310 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.753280 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-home\") pod \"distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.753448 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.753352 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-tmp-dir\") pod \"distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.753617 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.753587 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-model-cache\") pod \"distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.755128 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.755085 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-dshm\") pod \"distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.755631 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.755613 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aafa0b2-1b3f-479a-89a5-2041cf887a21-tls-certs\") pod \"distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.776687 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.776641 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2lvp\" (UniqueName: \"kubernetes.io/projected/3aafa0b2-1b3f-479a-89a5-2041cf887a21-kube-api-access-p2lvp\") pod \"distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:36.808051 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:36.808021 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:37.108973 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:37.108939 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d"] Oct 30 01:51:37.113845 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:51:37.113815 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3aafa0b2_1b3f_479a_89a5_2041cf887a21.slice/crio-0867b759325d046a77adaed49bc94ca1a5937502d1a4ace5fc98f87408f8fae4 WatchSource:0}: Error finding container 0867b759325d046a77adaed49bc94ca1a5937502d1a4ace5fc98f87408f8fae4: Status 404 returned error can't find the container with id 0867b759325d046a77adaed49bc94ca1a5937502d1a4ace5fc98f87408f8fae4 Oct 30 01:51:37.349180 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:37.349141 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" event={"ID":"3aafa0b2-1b3f-479a-89a5-2041cf887a21","Type":"ContainerStarted","Data":"f49533f7d96f11b6b6bba1a67e941e25792d99ae7e53fa4f8301932ab34249ca"} Oct 30 01:51:37.349360 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:37.349191 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" event={"ID":"3aafa0b2-1b3f-479a-89a5-2041cf887a21","Type":"ContainerStarted","Data":"0867b759325d046a77adaed49bc94ca1a5937502d1a4ace5fc98f87408f8fae4"} Oct 30 01:51:37.390352 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:37.390258 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podStartSLOduration=1.390246759 podStartE2EDuration="1.390246759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:37.389221624 +0000 UTC m=+1326.968012705" watchObservedRunningTime="2026-10-30 01:51:37.390246759 +0000 UTC m=+1326.969037839" Oct 30 01:51:37.809083 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:37.809043 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:38.032282 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:38.032247 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:39.576499 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:39.576455 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:40.244783 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:40.244745 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:41.283128 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:41.283035 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:42.793585 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:42.793547 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:44.285668 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:44.285631 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:44.729522 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:44.729484 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:51:46.270234 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:46.270199 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:46.809022 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:46.808982 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:46.826621 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:46.826594 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:46.911188 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:46.911153 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw"] Oct 30 01:51:46.915740 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:46.915715 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:46.929920 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:46.929895 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c40a89-openshift-default-dockercfg-vvdq4\"" Oct 30 01:51:46.944436 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:46.944406 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw"] Oct 30 01:51:46.959172 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:46.959133 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:46.959329 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:46.959184 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q97zk\" (UniqueName: \"kubernetes.io/projected/925ff5f4-100e-46a4-a906-3a9ecef96765-kube-api-access-q97zk\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:46.959329 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:46.959244 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:46.959430 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:46.959405 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:46.959556 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:46.959504 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:46.959652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:46.959584 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:46.959652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:46.959631 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:46.959768 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:46.959706 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925ff5f4-100e-46a4-a906-3a9ecef96765-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:46.959768 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:46.959763 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.061287 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.061191 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.061287 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.061248 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.061287 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.061286 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925ff5f4-100e-46a4-a906-3a9ecef96765-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.061613 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.061339 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.061613 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.061390 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.061613 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.061414 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q97zk\" (UniqueName: \"kubernetes.io/projected/925ff5f4-100e-46a4-a906-3a9ecef96765-kube-api-access-q97zk\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.061613 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.061472 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.061613 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.061514 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.061613 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.061553 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.061936 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.061808 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.061936 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.061882 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.062043 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.062008 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.062159 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.062135 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.062207 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.062180 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925ff5f4-100e-46a4-a906-3a9ecef96765-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.063635 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.063613 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.063876 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.063852 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.084641 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.084616 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.089057 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.089036 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q97zk\" (UniqueName: \"kubernetes.io/projected/925ff5f4-100e-46a4-a906-3a9ecef96765-kube-api-access-q97zk\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.229856 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.229813 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:47.437207 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.437173 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:51:47.553401 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.553364 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw"] Oct 30 01:51:47.556780 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:51:47.556754 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod925ff5f4_100e_46a4_a906_3a9ecef96765.slice/crio-045467e453ca173937713fcc7250644aebf2b99f6c897b3d78b593c7e6c2cbfd WatchSource:0}: Error finding container 045467e453ca173937713fcc7250644aebf2b99f6c897b3d78b593c7e6c2cbfd: Status 404 returned error can't find the container with id 045467e453ca173937713fcc7250644aebf2b99f6c897b3d78b593c7e6c2cbfd Oct 30 01:51:47.559047 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.559014 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:51:47.559177 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.559105 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:51:47.559177 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:47.559152 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:51:48.070727 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:48.070682 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:48.436436 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:48.436389 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" event={"ID":"925ff5f4-100e-46a4-a906-3a9ecef96765","Type":"ContainerStarted","Data":"3a91eb3f07c5ae57906745fbe42b211e39a26aee6823da7d6ef50d9355bdf9f7"} Oct 30 01:51:48.436642 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:48.436458 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" event={"ID":"925ff5f4-100e-46a4-a906-3a9ecef96765","Type":"ContainerStarted","Data":"045467e453ca173937713fcc7250644aebf2b99f6c897b3d78b593c7e6c2cbfd"} Oct 30 01:51:48.493911 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:48.493862 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" podStartSLOduration=2.493849448 podStartE2EDuration="2.493849448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:48.493713563 +0000 UTC m=+1338.072504644" watchObservedRunningTime="2026-10-30 01:51:48.493849448 +0000 UTC m=+1338.072640587" Oct 30 01:51:49.231287 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:49.231243 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:49.231870 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:49.231840 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 01:51:49.231938 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:49.231905 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" podUID="925ff5f4-100e-46a4-a906-3a9ecef96765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 01:51:49.683879 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:49.683845 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:50.230364 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:50.230325 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 01:51:50.230587 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:50.230404 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" podUID="925ff5f4-100e-46a4-a906-3a9ecef96765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 01:51:51.230815 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:51.230341 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 01:51:51.230815 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:51.230397 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" podUID="925ff5f4-100e-46a4-a906-3a9ecef96765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 01:51:51.280223 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:51.280191 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:52.231512 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:52.231462 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 01:51:52.231898 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:52.231527 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" podUID="925ff5f4-100e-46a4-a906-3a9ecef96765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 01:51:52.788085 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:52.788049 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:53.231111 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:53.230502 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 01:51:53.231111 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:53.230569 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" podUID="925ff5f4-100e-46a4-a906-3a9ecef96765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 01:51:54.230961 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.230918 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Oct 30 01:51:54.231371 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.230980 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" podUID="925ff5f4-100e-46a4-a906-3a9ecef96765" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Oct 30 01:51:54.379230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.379196 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:54.774896 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.774855 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25"] Oct 30 01:51:54.778714 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.778686 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.793251 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.793223 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-719029-openshift-default-dockercfg-9jg4v\"" Oct 30 01:51:54.810987 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.810951 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25"] Oct 30 01:51:54.880560 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.880528 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.880729 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.880584 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.880729 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.880631 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.880729 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.880695 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.880826 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.880806 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cfa231-6176-4512-9307-aa872cbd3f6a-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.880861 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.880825 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.880861 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.880844 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.880935 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.880878 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.881014 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.880998 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l9jv\" (UniqueName: \"kubernetes.io/projected/43cfa231-6176-4512-9307-aa872cbd3f6a-kube-api-access-6l9jv\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.981596 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.981555 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l9jv\" (UniqueName: \"kubernetes.io/projected/43cfa231-6176-4512-9307-aa872cbd3f6a-kube-api-access-6l9jv\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.981802 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.981641 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.981802 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.981686 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.981802 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.981707 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.981942 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.981827 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.981942 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.981908 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cfa231-6176-4512-9307-aa872cbd3f6a-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.982034 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.981937 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.982034 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.981966 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.982034 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.982005 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.982357 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.982334 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.982477 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.982407 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.982477 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.982465 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.982553 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.982537 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.982821 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.982798 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cfa231-6176-4512-9307-aa872cbd3f6a-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.984512 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.984491 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:54.984599 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:54.984582 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:55.004583 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.004549 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l9jv\" (UniqueName: \"kubernetes.io/projected/43cfa231-6176-4512-9307-aa872cbd3f6a-kube-api-access-6l9jv\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:55.004723 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.004704 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:55.093527 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.093438 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:55.225823 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:51:55.225747 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:51:55.235582 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.235546 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:55.236397 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.236343 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:55.236933 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.236916 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:51:55.432076 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.432041 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25"] Oct 30 01:51:55.437898 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:51:55.437856 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cfa231_6176_4512_9307_aa872cbd3f6a.slice/crio-e28bf9786859c97f1cd53db0e9329e739d50a9e56f1b7f73279333e69db22194 WatchSource:0}: Error finding container e28bf9786859c97f1cd53db0e9329e739d50a9e56f1b7f73279333e69db22194: Status 404 returned error can't find the container with id e28bf9786859c97f1cd53db0e9329e739d50a9e56f1b7f73279333e69db22194 Oct 30 01:51:55.441232 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.441173 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:51:55.441355 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.441267 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:51:55.441355 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.441343 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:51:55.482845 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.482809 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" event={"ID":"43cfa231-6176-4512-9307-aa872cbd3f6a","Type":"ContainerStarted","Data":"e28bf9786859c97f1cd53db0e9329e739d50a9e56f1b7f73279333e69db22194"} Oct 30 01:51:55.630142 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.630105 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7"] Oct 30 01:51:55.643684 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.643654 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.657193 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.657156 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7"] Oct 30 01:51:55.660806 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.660771 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-78d5ff-openshift-default-dockercfg-s66rj\"" Oct 30 01:51:55.790536 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.790437 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.790536 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.790507 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.790746 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.790602 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpkjp\" (UniqueName: \"kubernetes.io/projected/41479a83-7edc-4856-b62c-e12ee3c2d814-kube-api-access-qpkjp\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.790746 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.790637 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41479a83-7edc-4856-b62c-e12ee3c2d814-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.790746 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.790658 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.790868 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.790772 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.790911 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.790873 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.790955 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.790916 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.790955 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.790932 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.892425 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.892389 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.892609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.892485 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.892609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.892540 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.892609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.892564 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.892766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.892630 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.892766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.892662 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.892766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.892714 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpkjp\" (UniqueName: \"kubernetes.io/projected/41479a83-7edc-4856-b62c-e12ee3c2d814-kube-api-access-qpkjp\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.892766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.892748 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41479a83-7edc-4856-b62c-e12ee3c2d814-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.892971 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.892774 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.892971 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.892938 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.893067 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.893002 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.893231 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.893211 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.893389 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.893368 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.893578 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.893563 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41479a83-7edc-4856-b62c-e12ee3c2d814-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.894892 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.894872 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.895016 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.895002 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.915070 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.915038 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.915288 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.915267 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpkjp\" (UniqueName: \"kubernetes.io/projected/41479a83-7edc-4856-b62c-e12ee3c2d814-kube-api-access-qpkjp\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.961342 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.961311 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:55.992844 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:55.992812 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:56.294495 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:56.294463 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7"] Oct 30 01:51:56.298780 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:51:56.298739 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41479a83_7edc_4856_b62c_e12ee3c2d814.slice/crio-ddcc60c550a402178db2b8856e3984e073fcf1fa54331dcaa469b50c62de77fe WatchSource:0}: Error finding container ddcc60c550a402178db2b8856e3984e073fcf1fa54331dcaa469b50c62de77fe: Status 404 returned error can't find the container with id ddcc60c550a402178db2b8856e3984e073fcf1fa54331dcaa469b50c62de77fe Oct 30 01:51:56.312538 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:56.310664 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:51:56.312538 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:56.310756 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:51:56.312538 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:56.310796 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:51:56.489770 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:56.489735 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" event={"ID":"41479a83-7edc-4856-b62c-e12ee3c2d814","Type":"ContainerStarted","Data":"d64ee70885995a31b032b539791e26f2b8a07643aa3cce12f25b9723265588f0"} Oct 30 01:51:56.489935 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:56.489789 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" event={"ID":"41479a83-7edc-4856-b62c-e12ee3c2d814","Type":"ContainerStarted","Data":"ddcc60c550a402178db2b8856e3984e073fcf1fa54331dcaa469b50c62de77fe"} Oct 30 01:51:56.492814 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:56.492791 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" event={"ID":"43cfa231-6176-4512-9307-aa872cbd3f6a","Type":"ContainerStarted","Data":"126ec86eb84a33e929617f6a64ff358416602e33ddb9f14103ce4f5038173e70"} Oct 30 01:51:56.586654 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:56.586547 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" podStartSLOduration=1.586531613 podStartE2EDuration="1.586531613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:56.541395867 +0000 UTC m=+1346.120186948" watchObservedRunningTime="2026-10-30 01:51:56.586531613 +0000 UTC m=+1346.165322697" Oct 30 01:51:56.586867 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:56.586845 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" podStartSLOduration=2.586839176 podStartE2EDuration="2.586839176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:56.586163659 +0000 UTC m=+1346.164954740" watchObservedRunningTime="2026-10-30 01:51:56.586839176 +0000 UTC m=+1346.165630257" Oct 30 01:51:56.961817 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:56.961749 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:51:57.094581 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:57.094375 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:51:57.095790 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:57.095762 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 01:51:57.095912 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:57.095828 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" podUID="43cfa231-6176-4512-9307-aa872cbd3f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 01:51:57.716114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:57.712810 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:57.963811 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:57.963778 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 01:51:57.964043 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:57.964020 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" podUID="41479a83-7edc-4856-b62c-e12ee3c2d814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 01:51:58.094540 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:58.094457 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 01:51:58.094540 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:58.094506 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" podUID="43cfa231-6176-4512-9307-aa872cbd3f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 01:51:58.437113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:58.429288 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:58.962658 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:58.962609 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 01:51:58.963070 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:58.962695 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" podUID="41479a83-7edc-4856-b62c-e12ee3c2d814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 01:51:59.094506 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:59.094463 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 01:51:59.094711 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:59.094521 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" podUID="43cfa231-6176-4512-9307-aa872cbd3f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 01:51:59.563247 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:59.563211 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:51:59.961806 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:59.961766 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 01:51:59.962029 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:51:59.961828 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" podUID="41479a83-7edc-4856-b62c-e12ee3c2d814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 01:52:00.094689 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:00.094654 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 01:52:00.095072 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:00.094712 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" podUID="43cfa231-6176-4512-9307-aa872cbd3f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 01:52:00.961860 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:00.961818 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 01:52:00.962025 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:00.961876 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" podUID="41479a83-7edc-4856-b62c-e12ee3c2d814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 01:52:01.094433 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:01.094398 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 01:52:01.094637 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:01.094459 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" podUID="43cfa231-6176-4512-9307-aa872cbd3f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 01:52:01.297283 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:01.297200 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:01.962257 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:01.962221 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 01:52:01.962487 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:01.962278 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" podUID="41479a83-7edc-4856-b62c-e12ee3c2d814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 01:52:02.094042 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:02.093873 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 01:52:02.094042 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:02.093938 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" podUID="43cfa231-6176-4512-9307-aa872cbd3f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 01:52:02.752339 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:02.752306 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:02.962711 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:02.962669 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 30 01:52:02.962901 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:02.962739 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" podUID="41479a83-7edc-4856-b62c-e12ee3c2d814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 30 01:52:03.085527 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:03.085429 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:03.100280 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:03.100247 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:52:03.100644 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:03.100620 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:52:03.101627 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:03.101606 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 01:52:03.966169 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:03.966140 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:52:03.966585 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:03.966389 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:52:03.967034 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:03.967017 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 01:52:04.672416 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:04.672372 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:05.815785 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:05.815747 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:52:05.899610 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:05.899574 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt"] Oct 30 01:52:05.903709 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:05.903668 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:05.918714 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:05.918680 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-f6f4f0-openshift-default-dockercfg-jl28w\"" Oct 30 01:52:05.949233 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:05.949201 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt"] Oct 30 01:52:06.040898 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.040855 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.040898 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.040898 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.041177 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.041062 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.041234 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.041204 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.041293 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.041259 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.041351 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.041313 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.041351 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.041337 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.041456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.041372 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.041511 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.041463 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68wgw\" (UniqueName: \"kubernetes.io/projected/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-kube-api-access-68wgw\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.142180 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.142068 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68wgw\" (UniqueName: \"kubernetes.io/projected/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-kube-api-access-68wgw\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.142180 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.142172 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.142406 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.142200 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.142406 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.142238 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.142406 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.142297 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.142406 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.142383 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.142635 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.142411 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.142635 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.142444 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.142635 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.142501 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.142784 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.142748 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.142842 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.142792 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.142905 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.142837 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.143060 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.143038 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.143138 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.143060 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.144923 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.144900 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.145127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.145108 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.165014 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.164985 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68wgw\" (UniqueName: \"kubernetes.io/projected/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-kube-api-access-68wgw\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.165141 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.165012 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.221991 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.221954 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:06.283033 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.282991 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:06.555149 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.555076 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt"] Oct 30 01:52:06.560576 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:52:06.560541 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aca41c6_d766_49a1_8b29_6e55cd7cb8bc.slice/crio-38bc9859dbc2ee589fe27e13e6eae017858a87116c551afcee5f17edc3f68522 WatchSource:0}: Error finding container 38bc9859dbc2ee589fe27e13e6eae017858a87116c551afcee5f17edc3f68522: Status 404 returned error can't find the container with id 38bc9859dbc2ee589fe27e13e6eae017858a87116c551afcee5f17edc3f68522 Oct 30 01:52:06.563611 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.563572 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:52:06.563712 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.563657 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:52:06.563712 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.563697 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:52:06.615899 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:06.615865 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" event={"ID":"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc","Type":"ContainerStarted","Data":"38bc9859dbc2ee589fe27e13e6eae017858a87116c551afcee5f17edc3f68522"} Oct 30 01:52:07.648128 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:07.647209 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" event={"ID":"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc","Type":"ContainerStarted","Data":"3a734fa25185c308dc92d15f47078ab76d4e137255a5dea9af10a5b83172e6b2"} Oct 30 01:52:07.708111 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:07.707124 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" podStartSLOduration=2.7070856880000003 podStartE2EDuration="2.707085688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:07.704368865 +0000 UTC m=+1357.283159947" watchObservedRunningTime="2026-10-30 01:52:07.707085688 +0000 UTC m=+1357.285876771" Oct 30 01:52:07.997036 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:07.996996 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:08.239249 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:08.239211 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:08.239880 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:08.239851 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 01:52:08.240003 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:08.239903 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" podUID="6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 01:52:08.375185 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:08.374618 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:09.223021 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:09.222523 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 01:52:09.223021 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:09.222606 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" podUID="6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 01:52:09.795981 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:09.795941 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:10.222544 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:10.222502 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 01:52:10.222859 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:10.222829 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" podUID="6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 01:52:11.227679 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:11.223502 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 01:52:11.227679 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:11.223567 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" podUID="6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 01:52:11.473145 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:11.473111 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:12.222571 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:12.222479 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 01:52:12.222868 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:12.222840 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" podUID="6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 01:52:13.092227 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:13.092188 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:13.223318 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:13.223273 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 30 01:52:13.223525 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:13.223352 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" podUID="6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 30 01:52:14.227725 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:14.227686 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:14.228230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:14.228181 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:14.228821 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:14.228798 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 01:52:14.741637 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:14.741596 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:16.723303 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:16.723258 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:18.278997 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:18.278967 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:19.831302 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:19.831257 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:20.195950 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:20.195913 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:21.559201 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:21.559108 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:23.064458 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:23.064388 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:24.671472 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:24.671440 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:25.221640 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:52:25.221593 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:52:26.171066 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:26.171029 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:27.792585 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:27.792529 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:28.088509 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:28.088413 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:52:29.689332 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:29.689296 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:31.499575 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:31.499536 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:32.065789 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.065752 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt"] Oct 30 01:52:32.072377 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.072349 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.108121 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.108073 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt"] Oct 30 01:52:32.183605 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.183555 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.183779 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.183634 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.183779 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.183698 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.183779 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.183741 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2wgh\" (UniqueName: \"kubernetes.io/projected/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-kube-api-access-d2wgh\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.183779 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.183767 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.183915 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.183830 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.183951 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.183924 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.184003 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.183984 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.184065 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.184042 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.285039 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.285001 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.285249 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.285062 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.285249 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.285105 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.285249 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.285179 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.285249 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.285233 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.285474 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.285269 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.285474 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.285294 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2wgh\" (UniqueName: \"kubernetes.io/projected/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-kube-api-access-d2wgh\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.285474 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.285327 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.285474 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.285353 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.285673 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.285530 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.285727 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.285709 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.285841 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.285815 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.285914 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.285893 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.286329 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.286310 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.287997 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.287971 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.288323 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.288300 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.308059 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.308019 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.308219 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.308137 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2wgh\" (UniqueName: \"kubernetes.io/projected/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-kube-api-access-d2wgh\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.386698 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.386601 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:32.714658 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.714615 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt"] Oct 30 01:52:32.725527 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:52:32.725479 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2123b7_6e81_4dce_8893_29ec86f7bd7e.slice/crio-a506192fa53bf73cdce56952358a734d810377903e5fc3b4a20d58ae28fe8f91 WatchSource:0}: Error finding container a506192fa53bf73cdce56952358a734d810377903e5fc3b4a20d58ae28fe8f91: Status 404 returned error can't find the container with id a506192fa53bf73cdce56952358a734d810377903e5fc3b4a20d58ae28fe8f91 Oct 30 01:52:32.728425 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.728387 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:52:32.728530 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.728473 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:52:32.728530 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.728519 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:52:32.819822 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.819780 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" event={"ID":"7d2123b7-6e81-4dce-8893-29ec86f7bd7e","Type":"ContainerStarted","Data":"1877cd29a4d898fa9bdb654f7b7186e6a62638bfcaf99503336cc1ae993c0470"} Oct 30 01:52:32.819982 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.819852 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" event={"ID":"7d2123b7-6e81-4dce-8893-29ec86f7bd7e","Type":"ContainerStarted","Data":"a506192fa53bf73cdce56952358a734d810377903e5fc3b4a20d58ae28fe8f91"} Oct 30 01:52:32.864976 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:32.864912 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" podStartSLOduration=0.864892161 podStartE2EDuration="864.892161ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:32.863398918 +0000 UTC m=+1382.442190023" watchObservedRunningTime="2026-10-30 01:52:32.864892161 +0000 UTC m=+1382.443683243" Oct 30 01:52:33.286060 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:33.286026 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:33.386870 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:33.386834 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:33.388692 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:33.388661 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:52:33.388840 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:33.388719 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" podUID="7d2123b7-6e81-4dce-8893-29ec86f7bd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 01:52:34.218116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:34.218065 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:34.387239 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:34.387207 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:52:34.387397 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:34.387273 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" podUID="7d2123b7-6e81-4dce-8893-29ec86f7bd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 01:52:35.031684 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:35.031644 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:35.147535 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:35.147480 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:35.387459 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:35.387369 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:52:35.387459 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:35.387436 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" podUID="7d2123b7-6e81-4dce-8893-29ec86f7bd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 01:52:35.651027 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:35.650935 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:36.387582 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:36.387542 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:52:36.388125 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:36.387621 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" podUID="7d2123b7-6e81-4dce-8893-29ec86f7bd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 01:52:36.889185 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:36.889137 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:37.121074 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.121036 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x"] Oct 30 01:52:37.137922 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.137890 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x" Oct 30 01:52:37.140926 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.140852 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x"] Oct 30 01:52:37.166916 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.166890 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:52:37.167143 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.166950 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:52:37.167143 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.166997 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:52:37.167366 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.167051 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:52:37.233966 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.233916 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:37.273663 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.273626 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac29fe5-9261-48a6-9b20-d9a192087501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f0cc884-dm48x\" (UID: \"aac29fe5-9261-48a6-9b20-d9a192087501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x" Oct 30 01:52:37.273663 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.273673 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac29fe5-9261-48a6-9b20-d9a192087501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f0cc884-dm48x\" (UID: \"aac29fe5-9261-48a6-9b20-d9a192087501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x" Oct 30 01:52:37.273947 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.273705 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvrkp\" (UniqueName: \"kubernetes.io/projected/aac29fe5-9261-48a6-9b20-d9a192087501-kube-api-access-qvrkp\") pod \"maas-api-revoke-sub-4f0cc884-dm48x\" (UID: \"aac29fe5-9261-48a6-9b20-d9a192087501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x" Oct 30 01:52:37.375575 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.375495 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac29fe5-9261-48a6-9b20-d9a192087501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f0cc884-dm48x\" (UID: \"aac29fe5-9261-48a6-9b20-d9a192087501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x" Oct 30 01:52:37.376445 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.376412 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvrkp\" (UniqueName: \"kubernetes.io/projected/aac29fe5-9261-48a6-9b20-d9a192087501-kube-api-access-qvrkp\") pod \"maas-api-revoke-sub-4f0cc884-dm48x\" (UID: \"aac29fe5-9261-48a6-9b20-d9a192087501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x" Oct 30 01:52:37.376884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.376862 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac29fe5-9261-48a6-9b20-d9a192087501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f0cc884-dm48x\" (UID: \"aac29fe5-9261-48a6-9b20-d9a192087501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x" Oct 30 01:52:37.377759 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.377728 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac29fe5-9261-48a6-9b20-d9a192087501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f0cc884-dm48x\" (UID: \"aac29fe5-9261-48a6-9b20-d9a192087501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x" Oct 30 01:52:37.387539 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.387509 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:52:37.387798 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.387773 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" podUID="7d2123b7-6e81-4dce-8893-29ec86f7bd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 01:52:37.391883 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.391821 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac29fe5-9261-48a6-9b20-d9a192087501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f0cc884-dm48x\" (UID: \"aac29fe5-9261-48a6-9b20-d9a192087501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x" Oct 30 01:52:37.416221 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.416188 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvrkp\" (UniqueName: \"kubernetes.io/projected/aac29fe5-9261-48a6-9b20-d9a192087501-kube-api-access-qvrkp\") pod \"maas-api-revoke-sub-4f0cc884-dm48x\" (UID: \"aac29fe5-9261-48a6-9b20-d9a192087501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x" Oct 30 01:52:37.466522 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.461280 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x" Oct 30 01:52:37.827135 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.827069 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x"] Oct 30 01:52:37.872802 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:37.872772 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x" event={"ID":"aac29fe5-9261-48a6-9b20-d9a192087501","Type":"ContainerStarted","Data":"acd0eb8878174d93192879afbe364d12c83ad8e8eb06ddc9c8968b9b5a299c9a"} Oct 30 01:52:38.387922 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:38.387881 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:52:38.388411 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:38.387941 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" podUID="7d2123b7-6e81-4dce-8893-29ec86f7bd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 01:52:38.613609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:38.613484 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:38.831531 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:38.831493 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg"] Oct 30 01:52:38.886995 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:38.886907 5998 generic.go:356] "Generic (PLEG): container finished" podID="aac29fe5-9261-48a6-9b20-d9a192087501" containerID="bddde80c16b650abd930a4e97430f726f4ccd5c274455feda7514bb632ceed9f" exitCode=0 Oct 30 01:52:38.887275 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:38.886953 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x" event={"ID":"aac29fe5-9261-48a6-9b20-d9a192087501","Type":"ContainerDied","Data":"bddde80c16b650abd930a4e97430f726f4ccd5c274455feda7514bb632ceed9f"} Oct 30 01:52:38.924106 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:38.924056 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:39.387648 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:39.387612 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:52:39.387828 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:39.387668 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" podUID="7d2123b7-6e81-4dce-8893-29ec86f7bd7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Oct 30 01:52:39.924901 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:39.924858 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677"] Oct 30 01:52:39.930417 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:39.930384 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:39.979389 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:39.966768 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677"] Oct 30 01:52:40.014297 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.014255 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.014500 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.014465 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpp4x\" (UniqueName: \"kubernetes.io/projected/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-kube-api-access-mpp4x\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.014575 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.014518 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.014630 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.014571 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.014702 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.014681 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.014758 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.014742 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.014806 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.014777 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.014859 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.014843 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.014911 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.014884 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.058579 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.058543 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x" Oct 30 01:52:40.116172 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.116122 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.116359 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.116204 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.116359 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.116338 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpp4x\" (UniqueName: \"kubernetes.io/projected/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-kube-api-access-mpp4x\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.116481 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.116383 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.116481 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.116421 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.116481 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.116472 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.116652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.116523 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.116652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.116551 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.116652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.116585 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.117966 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.117849 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.117966 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.117912 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.118239 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.118143 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.118381 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.118351 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.118629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.118603 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.120046 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.120023 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.120389 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.120368 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.151946 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.151613 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.151946 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.151848 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpp4x\" (UniqueName: \"kubernetes.io/projected/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-kube-api-access-mpp4x\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-5p677\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.217717 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.217676 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aac29fe5-9261-48a6-9b20-d9a192087501-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac29fe5-9261-48a6-9b20-d9a192087501-maas-api-openshift-ca\") pod \"aac29fe5-9261-48a6-9b20-d9a192087501\" (UID: \"aac29fe5-9261-48a6-9b20-d9a192087501\") " Oct 30 01:52:40.217907 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.217772 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aac29fe5-9261-48a6-9b20-d9a192087501-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac29fe5-9261-48a6-9b20-d9a192087501-maas-api-xks-ca\") pod \"aac29fe5-9261-48a6-9b20-d9a192087501\" (UID: \"aac29fe5-9261-48a6-9b20-d9a192087501\") " Oct 30 01:52:40.217907 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.217826 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac29fe5-9261-48a6-9b20-d9a192087501-kube-api-access-qvrkp\" (UniqueName: \"kubernetes.io/projected/aac29fe5-9261-48a6-9b20-d9a192087501-kube-api-access-qvrkp\") pod \"aac29fe5-9261-48a6-9b20-d9a192087501\" (UID: \"aac29fe5-9261-48a6-9b20-d9a192087501\") " Oct 30 01:52:40.218172 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.218141 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac29fe5-9261-48a6-9b20-d9a192087501-maas-api-openshift-ca" pod "aac29fe5-9261-48a6-9b20-d9a192087501" (UID: "aac29fe5-9261-48a6-9b20-d9a192087501"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:40.218462 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.218438 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aac29fe5-9261-48a6-9b20-d9a192087501-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:52:40.219990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.219964 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac29fe5-9261-48a6-9b20-d9a192087501-kube-api-access-qvrkp" pod "aac29fe5-9261-48a6-9b20-d9a192087501" (UID: "aac29fe5-9261-48a6-9b20-d9a192087501"). InnerVolumeSpecName "kube-api-access-qvrkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:40.220299 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.220278 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac29fe5-9261-48a6-9b20-d9a192087501-maas-api-xks-ca" pod "aac29fe5-9261-48a6-9b20-d9a192087501" (UID: "aac29fe5-9261-48a6-9b20-d9a192087501"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:52:40.246281 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.246255 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:40.319905 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.319873 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aac29fe5-9261-48a6-9b20-d9a192087501-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:52:40.319905 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.319902 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvrkp\" (UniqueName: \"kubernetes.io/projected/aac29fe5-9261-48a6-9b20-d9a192087501-kube-api-access-qvrkp\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:52:40.392901 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.392871 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:40.393306 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.393283 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:40.394140 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.394118 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 01:52:40.552507 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.552476 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:40.602637 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.602603 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677"] Oct 30 01:52:40.606052 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:52:40.606017 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592f8d79_2a33_4c5b_9639_c60bd7aedf9f.slice/crio-ae27310ee77b367dcb73d0ff45d2d21a237ece416106b1365faa3a76df4fdd84 WatchSource:0}: Error finding container ae27310ee77b367dcb73d0ff45d2d21a237ece416106b1365faa3a76df4fdd84: Status 404 returned error can't find the container with id ae27310ee77b367dcb73d0ff45d2d21a237ece416106b1365faa3a76df4fdd84 Oct 30 01:52:40.608496 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.608459 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:52:40.608614 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.608544 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:52:40.608614 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.608589 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:52:40.903937 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.903840 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" event={"ID":"592f8d79-2a33-4c5b-9639-c60bd7aedf9f","Type":"ContainerStarted","Data":"de7805abc6adf67ef4d005dd78bc212f8e532d9e870b77cbb3cb0acb31c5bf38"} Oct 30 01:52:40.903937 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.903914 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" event={"ID":"592f8d79-2a33-4c5b-9639-c60bd7aedf9f","Type":"ContainerStarted","Data":"ae27310ee77b367dcb73d0ff45d2d21a237ece416106b1365faa3a76df4fdd84"} Oct 30 01:52:40.905635 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.905572 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x" event={"ID":"aac29fe5-9261-48a6-9b20-d9a192087501","Type":"ContainerDied","Data":"acd0eb8878174d93192879afbe364d12c83ad8e8eb06ddc9c8968b9b5a299c9a"} Oct 30 01:52:40.905755 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.905675 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acd0eb8878174d93192879afbe364d12c83ad8e8eb06ddc9c8968b9b5a299c9a" Oct 30 01:52:40.905755 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.905613 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x" Oct 30 01:52:40.967994 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:40.967942 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" podStartSLOduration=1.96792781 podStartE2EDuration="1.96792781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:40.966580572 +0000 UTC m=+1390.545371653" watchObservedRunningTime="2026-10-30 01:52:40.96792781 +0000 UTC m=+1390.546718968" Oct 30 01:52:41.246487 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:41.246440 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:41.248100 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:41.248064 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 01:52:41.248250 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:41.248157 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" podUID="592f8d79-2a33-4c5b-9639-c60bd7aedf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 01:52:42.247353 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:42.247308 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 01:52:42.247849 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:42.247381 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" podUID="592f8d79-2a33-4c5b-9639-c60bd7aedf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 01:52:43.247372 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:43.247332 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 01:52:43.247752 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:43.247408 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" podUID="592f8d79-2a33-4c5b-9639-c60bd7aedf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 01:52:43.643147 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:43.642990 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:43.875066 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:43.875032 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:44.246903 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:44.246870 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 01:52:44.247110 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:44.246941 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" podUID="592f8d79-2a33-4c5b-9639-c60bd7aedf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 01:52:45.247287 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:45.247245 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 01:52:45.247705 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:45.247316 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" podUID="592f8d79-2a33-4c5b-9639-c60bd7aedf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 01:52:45.452309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:45.452272 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:46.248117 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:46.247643 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 01:52:46.248117 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:46.247704 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" podUID="592f8d79-2a33-4c5b-9639-c60bd7aedf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 01:52:46.965479 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:46.965447 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:47.456958 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.456922 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8"] Oct 30 01:52:47.457645 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.457627 5998 cpu_manager.go:414] "containerMap: removing container" podUID="aac29fe5-9261-48a6-9b20-d9a192087501" containerName="revoke-keys" Oct 30 01:52:47.457701 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.457648 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac29fe5-9261-48a6-9b20-d9a192087501" containerName="revoke-keys" Oct 30 01:52:47.457760 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.457748 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac29fe5-9261-48a6-9b20-d9a192087501" containerName="revoke-keys" Oct 30 01:52:47.461567 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.461539 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8" Oct 30 01:52:47.478058 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.478032 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:52:47.483222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.483195 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8"] Oct 30 01:52:47.490275 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.490160 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:52:47.490491 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.490207 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:52:47.490591 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.490233 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:52:47.609502 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.609463 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cjx7\" (UniqueName: \"kubernetes.io/projected/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-kube-api-access-5cjx7\") pod \"maas-api-revoke-sub-ca5ed0bf-bzhp8\" (UID: \"32f8b0ef-77f6-441e-ac51-a5e6fccbb750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8" Oct 30 01:52:47.609714 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.609590 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca5ed0bf-bzhp8\" (UID: \"32f8b0ef-77f6-441e-ac51-a5e6fccbb750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8" Oct 30 01:52:47.609714 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.609645 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca5ed0bf-bzhp8\" (UID: \"32f8b0ef-77f6-441e-ac51-a5e6fccbb750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8" Oct 30 01:52:47.634225 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.632764 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:47.634225 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.634148 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:47.635193 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.635167 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 01:52:47.710784 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.710691 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca5ed0bf-bzhp8\" (UID: \"32f8b0ef-77f6-441e-ac51-a5e6fccbb750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8" Oct 30 01:52:47.710784 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.710763 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca5ed0bf-bzhp8\" (UID: \"32f8b0ef-77f6-441e-ac51-a5e6fccbb750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8" Oct 30 01:52:47.711061 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.711041 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cjx7\" (UniqueName: \"kubernetes.io/projected/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-kube-api-access-5cjx7\") pod \"maas-api-revoke-sub-ca5ed0bf-bzhp8\" (UID: \"32f8b0ef-77f6-441e-ac51-a5e6fccbb750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8" Oct 30 01:52:47.712778 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.712748 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca5ed0bf-bzhp8\" (UID: \"32f8b0ef-77f6-441e-ac51-a5e6fccbb750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8" Oct 30 01:52:47.714469 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.714430 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca5ed0bf-bzhp8\" (UID: \"32f8b0ef-77f6-441e-ac51-a5e6fccbb750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8" Oct 30 01:52:47.734632 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.734595 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cjx7\" (UniqueName: \"kubernetes.io/projected/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-kube-api-access-5cjx7\") pod \"maas-api-revoke-sub-ca5ed0bf-bzhp8\" (UID: \"32f8b0ef-77f6-441e-ac51-a5e6fccbb750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8" Oct 30 01:52:47.779658 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:47.779629 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8" Oct 30 01:52:48.071869 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:48.071790 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:52:48.099559 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:48.099518 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8"] Oct 30 01:52:48.106037 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:52:48.105998 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f8b0ef_77f6_441e_ac51_a5e6fccbb750.slice/crio-36692a95d38357eab94b1f80dc873c3d2dbc0c16210034b33759d77fe8c1bee1 WatchSource:0}: Error finding container 36692a95d38357eab94b1f80dc873c3d2dbc0c16210034b33759d77fe8c1bee1: Status 404 returned error can't find the container with id 36692a95d38357eab94b1f80dc873c3d2dbc0c16210034b33759d77fe8c1bee1 Oct 30 01:52:48.619928 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:48.619892 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:48.999110 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:48.993384 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8" event={"ID":"32f8b0ef-77f6-441e-ac51-a5e6fccbb750","Type":"ContainerStarted","Data":"6d2d7914acc6013698a609befc657f2842d149174916bcd8aa32051eb19d6bbe"} Oct 30 01:52:48.999110 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:48.993497 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8" event={"ID":"32f8b0ef-77f6-441e-ac51-a5e6fccbb750","Type":"ContainerStarted","Data":"36692a95d38357eab94b1f80dc873c3d2dbc0c16210034b33759d77fe8c1bee1"} Oct 30 01:52:49.042365 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:49.041268 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8" podStartSLOduration=2.041234137 podStartE2EDuration="2.041234137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:49.037957887 +0000 UTC m=+1398.616748971" watchObservedRunningTime="2026-10-30 01:52:49.041234137 +0000 UTC m=+1398.620025219" Oct 30 01:52:50.010284 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:50.009680 5998 generic.go:356] "Generic (PLEG): container finished" podID="32f8b0ef-77f6-441e-ac51-a5e6fccbb750" containerID="6d2d7914acc6013698a609befc657f2842d149174916bcd8aa32051eb19d6bbe" exitCode=0 Oct 30 01:52:50.010284 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:50.009857 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8" event={"ID":"32f8b0ef-77f6-441e-ac51-a5e6fccbb750","Type":"ContainerDied","Data":"6d2d7914acc6013698a609befc657f2842d149174916bcd8aa32051eb19d6bbe"} Oct 30 01:52:51.061917 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.061885 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx"] Oct 30 01:52:51.066024 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.066003 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.090021 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.089777 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx"] Oct 30 01:52:51.191383 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.191342 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:51.198668 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.198641 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.198796 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.198698 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.198882 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.198850 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.199169 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.198939 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.199169 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.198981 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.199169 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.199052 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.199371 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.199167 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.199371 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.199223 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxmqr\" (UniqueName: \"kubernetes.io/projected/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-kube-api-access-sxmqr\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.199371 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.199306 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.208564 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.208536 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8" Oct 30 01:52:51.299827 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.299742 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-maas-api-xks-ca\") pod \"32f8b0ef-77f6-441e-ac51-a5e6fccbb750\" (UID: \"32f8b0ef-77f6-441e-ac51-a5e6fccbb750\") " Oct 30 01:52:51.299984 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.299874 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-maas-api-openshift-ca\") pod \"32f8b0ef-77f6-441e-ac51-a5e6fccbb750\" (UID: \"32f8b0ef-77f6-441e-ac51-a5e6fccbb750\") " Oct 30 01:52:51.299984 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.299926 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-kube-api-access-5cjx7\" (UniqueName: \"kubernetes.io/projected/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-kube-api-access-5cjx7\") pod \"32f8b0ef-77f6-441e-ac51-a5e6fccbb750\" (UID: \"32f8b0ef-77f6-441e-ac51-a5e6fccbb750\") " Oct 30 01:52:51.300153 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.300125 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.300209 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.300167 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.300256 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.300208 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.300256 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.300246 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.300342 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.300301 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxmqr\" (UniqueName: \"kubernetes.io/projected/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-kube-api-access-sxmqr\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.300403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.300348 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-maas-api-openshift-ca" pod "32f8b0ef-77f6-441e-ac51-a5e6fccbb750" (UID: "32f8b0ef-77f6-441e-ac51-a5e6fccbb750"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:51.300403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.300385 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.300508 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.300463 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.300563 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.300527 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.300622 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.300560 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.300674 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.300641 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:52:51.300941 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.300917 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.301018 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.300940 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.301077 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.301023 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.301242 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.301218 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.301628 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.301595 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.303471 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.303433 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-kube-api-access-5cjx7" pod "32f8b0ef-77f6-441e-ac51-a5e6fccbb750" (UID: "32f8b0ef-77f6-441e-ac51-a5e6fccbb750"). InnerVolumeSpecName "kube-api-access-5cjx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:51.303767 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.303744 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.304466 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.304435 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.316492 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.316151 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-maas-api-xks-ca" pod "32f8b0ef-77f6-441e-ac51-a5e6fccbb750" (UID: "32f8b0ef-77f6-441e-ac51-a5e6fccbb750"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:52:51.323885 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.323844 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.325054 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.325031 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxmqr\" (UniqueName: \"kubernetes.io/projected/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-kube-api-access-sxmqr\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.372194 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.372157 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8"] Oct 30 01:52:51.372698 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.372683 5998 cpu_manager.go:414] "containerMap: removing container" podUID="32f8b0ef-77f6-441e-ac51-a5e6fccbb750" containerName="revoke-keys" Oct 30 01:52:51.372758 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.372701 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f8b0ef-77f6-441e-ac51-a5e6fccbb750" containerName="revoke-keys" Oct 30 01:52:51.372817 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.372807 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f8b0ef-77f6-441e-ac51-a5e6fccbb750" containerName="revoke-keys" Oct 30 01:52:51.377347 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.377326 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8" Oct 30 01:52:51.381988 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.381961 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:51.400335 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.400306 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8"] Oct 30 01:52:51.401461 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.401440 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:52:51.401570 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.401464 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cjx7\" (UniqueName: \"kubernetes.io/projected/32f8b0ef-77f6-441e-ac51-a5e6fccbb750-kube-api-access-5cjx7\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:52:51.502398 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.502358 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8b7b91-c3df-4b25-9eb0-02dad493b607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df4b9a47-57ff8\" (UID: \"fb8b7b91-c3df-4b25-9eb0-02dad493b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8" Oct 30 01:52:51.502592 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.502443 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8b7b91-c3df-4b25-9eb0-02dad493b607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df4b9a47-57ff8\" (UID: \"fb8b7b91-c3df-4b25-9eb0-02dad493b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8" Oct 30 01:52:51.502592 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.502492 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsphv\" (UniqueName: \"kubernetes.io/projected/fb8b7b91-c3df-4b25-9eb0-02dad493b607-kube-api-access-lsphv\") pod \"maas-api-revoke-sub-df4b9a47-57ff8\" (UID: \"fb8b7b91-c3df-4b25-9eb0-02dad493b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8" Oct 30 01:52:51.603174 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.603054 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8b7b91-c3df-4b25-9eb0-02dad493b607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df4b9a47-57ff8\" (UID: \"fb8b7b91-c3df-4b25-9eb0-02dad493b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8" Oct 30 01:52:51.603174 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.603162 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8b7b91-c3df-4b25-9eb0-02dad493b607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df4b9a47-57ff8\" (UID: \"fb8b7b91-c3df-4b25-9eb0-02dad493b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8" Oct 30 01:52:51.603416 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.603222 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsphv\" (UniqueName: \"kubernetes.io/projected/fb8b7b91-c3df-4b25-9eb0-02dad493b607-kube-api-access-lsphv\") pod \"maas-api-revoke-sub-df4b9a47-57ff8\" (UID: \"fb8b7b91-c3df-4b25-9eb0-02dad493b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8" Oct 30 01:52:51.603737 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.603715 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8b7b91-c3df-4b25-9eb0-02dad493b607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df4b9a47-57ff8\" (UID: \"fb8b7b91-c3df-4b25-9eb0-02dad493b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8" Oct 30 01:52:51.605692 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.605664 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8b7b91-c3df-4b25-9eb0-02dad493b607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df4b9a47-57ff8\" (UID: \"fb8b7b91-c3df-4b25-9eb0-02dad493b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8" Oct 30 01:52:51.626625 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.626594 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsphv\" (UniqueName: \"kubernetes.io/projected/fb8b7b91-c3df-4b25-9eb0-02dad493b607-kube-api-access-lsphv\") pod \"maas-api-revoke-sub-df4b9a47-57ff8\" (UID: \"fb8b7b91-c3df-4b25-9eb0-02dad493b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8" Oct 30 01:52:51.695801 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.695773 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8" Oct 30 01:52:51.743267 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.743031 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx"] Oct 30 01:52:51.747564 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:52:51.747531 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34d09e7_ec9d_4160_9c74_13b2ccf8d871.slice/crio-2482b413756599445b0c5c358b5cd336c6f1cfc9f32718d500435702cb5ee18a WatchSource:0}: Error finding container 2482b413756599445b0c5c358b5cd336c6f1cfc9f32718d500435702cb5ee18a: Status 404 returned error can't find the container with id 2482b413756599445b0c5c358b5cd336c6f1cfc9f32718d500435702cb5ee18a Oct 30 01:52:51.750293 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.750257 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:52:51.750403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.750323 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:52:51.750403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:51.750361 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:52:52.014814 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:52.014773 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8"] Oct 30 01:52:52.027401 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:52.027371 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" event={"ID":"e34d09e7-ec9d-4160-9c74-13b2ccf8d871","Type":"ContainerStarted","Data":"b8c2f3f0f63a3531d1e6b740ccfc756a7c82191b4d723e0a46e3fc30f8d37830"} Oct 30 01:52:52.027505 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:52.027436 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" event={"ID":"e34d09e7-ec9d-4160-9c74-13b2ccf8d871","Type":"ContainerStarted","Data":"2482b413756599445b0c5c358b5cd336c6f1cfc9f32718d500435702cb5ee18a"} Oct 30 01:52:52.029355 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:52.029329 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8" event={"ID":"32f8b0ef-77f6-441e-ac51-a5e6fccbb750","Type":"ContainerDied","Data":"36692a95d38357eab94b1f80dc873c3d2dbc0c16210034b33759d77fe8c1bee1"} Oct 30 01:52:52.029463 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:52.029380 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36692a95d38357eab94b1f80dc873c3d2dbc0c16210034b33759d77fe8c1bee1" Oct 30 01:52:52.029463 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:52.029337 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8" Oct 30 01:52:52.077348 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:52.077280 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" podStartSLOduration=1.077258902 podStartE2EDuration="1.077258902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:52.071201044 +0000 UTC m=+1401.649992146" watchObservedRunningTime="2026-10-30 01:52:52.077258902 +0000 UTC m=+1401.656049983" Oct 30 01:52:52.382657 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:52.382628 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:52.384234 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:52.384203 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 01:52:52.384370 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:52.384262 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" podUID="e34d09e7-ec9d-4160-9c74-13b2ccf8d871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 01:52:52.806002 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:52.805952 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:53.038878 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:53.038835 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8" event={"ID":"fb8b7b91-c3df-4b25-9eb0-02dad493b607","Type":"ContainerStarted","Data":"c8be907f199dcb80db3186b1dbcb0c712aff44dab624456a132eeae19bd35717"} Oct 30 01:52:53.039058 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:53.038902 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8" event={"ID":"fb8b7b91-c3df-4b25-9eb0-02dad493b607","Type":"ContainerStarted","Data":"0d8acb1dccbd1d5426c55f4bc3ca8cab9ed9bc941cb5ff5b6d19167b9bbde3cd"} Oct 30 01:52:53.091432 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:53.091324 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8" podStartSLOduration=2.091310517 podStartE2EDuration="2.091310517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:53.090483875 +0000 UTC m=+1402.669274957" watchObservedRunningTime="2026-10-30 01:52:53.091310517 +0000 UTC m=+1402.670101598" Oct 30 01:52:53.382916 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:53.382818 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 01:52:53.382916 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:53.382882 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" podUID="e34d09e7-ec9d-4160-9c74-13b2ccf8d871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 01:52:53.861468 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:53.861422 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" podUID="56e784e5-a048-4f8e-acec-0eeae021fcfb" containerName="main" containerID="cri-o://75e072709ae61fe833f5fc2892320bac75235c6067b3be6cfd4cd20bdabc93ac" gracePeriod=45 Oct 30 01:52:54.048595 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.048561 5998 generic.go:356] "Generic (PLEG): container finished" podID="56e784e5-a048-4f8e-acec-0eeae021fcfb" containerID="75e072709ae61fe833f5fc2892320bac75235c6067b3be6cfd4cd20bdabc93ac" exitCode=0 Oct 30 01:52:54.048769 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.048665 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" event={"ID":"56e784e5-a048-4f8e-acec-0eeae021fcfb","Type":"ContainerDied","Data":"75e072709ae61fe833f5fc2892320bac75235c6067b3be6cfd4cd20bdabc93ac"} Oct 30 01:52:54.051440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.050534 5998 generic.go:356] "Generic (PLEG): container finished" podID="fb8b7b91-c3df-4b25-9eb0-02dad493b607" containerID="c8be907f199dcb80db3186b1dbcb0c712aff44dab624456a132eeae19bd35717" exitCode=0 Oct 30 01:52:54.051440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.051248 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8" event={"ID":"fb8b7b91-c3df-4b25-9eb0-02dad493b607","Type":"ContainerDied","Data":"c8be907f199dcb80db3186b1dbcb0c712aff44dab624456a132eeae19bd35717"} Oct 30 01:52:54.245539 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.245517 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:52:54.246658 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.246626 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-home\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-home\") pod \"56e784e5-a048-4f8e-acec-0eeae021fcfb\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " Oct 30 01:52:54.246957 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.246937 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e784e5-a048-4f8e-acec-0eeae021fcfb-kube-api-access-gfmgq\" (UniqueName: \"kubernetes.io/projected/56e784e5-a048-4f8e-acec-0eeae021fcfb-kube-api-access-gfmgq\") pod \"56e784e5-a048-4f8e-acec-0eeae021fcfb\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " Oct 30 01:52:54.247205 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.247019 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-home" pod "56e784e5-a048-4f8e-acec-0eeae021fcfb" (UID: "56e784e5-a048-4f8e-acec-0eeae021fcfb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:54.247662 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.247648 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:52:54.249292 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.249269 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e784e5-a048-4f8e-acec-0eeae021fcfb-kube-api-access-gfmgq" pod "56e784e5-a048-4f8e-acec-0eeae021fcfb" (UID: "56e784e5-a048-4f8e-acec-0eeae021fcfb"). InnerVolumeSpecName "kube-api-access-gfmgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:54.351463 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.350409 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-dshm\") pod \"56e784e5-a048-4f8e-acec-0eeae021fcfb\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " Oct 30 01:52:54.351463 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.350447 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56e784e5-a048-4f8e-acec-0eeae021fcfb-tls-certs\" (UniqueName: \"kubernetes.io/secret/56e784e5-a048-4f8e-acec-0eeae021fcfb-tls-certs\") pod \"56e784e5-a048-4f8e-acec-0eeae021fcfb\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " Oct 30 01:52:54.351463 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.350676 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-model-cache\") pod \"56e784e5-a048-4f8e-acec-0eeae021fcfb\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " Oct 30 01:52:54.351463 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.350725 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-tmp-dir\") pod \"56e784e5-a048-4f8e-acec-0eeae021fcfb\" (UID: \"56e784e5-a048-4f8e-acec-0eeae021fcfb\") " Oct 30 01:52:54.351463 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.351026 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-tmp-dir" pod "56e784e5-a048-4f8e-acec-0eeae021fcfb" (UID: "56e784e5-a048-4f8e-acec-0eeae021fcfb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:54.351463 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.351311 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfmgq\" (UniqueName: \"kubernetes.io/projected/56e784e5-a048-4f8e-acec-0eeae021fcfb-kube-api-access-gfmgq\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:52:54.351463 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.351320 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-model-cache" pod "56e784e5-a048-4f8e-acec-0eeae021fcfb" (UID: "56e784e5-a048-4f8e-acec-0eeae021fcfb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:54.351463 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.351334 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:52:54.353257 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.353234 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56e784e5-a048-4f8e-acec-0eeae021fcfb-tls-certs" pod "56e784e5-a048-4f8e-acec-0eeae021fcfb" (UID: "56e784e5-a048-4f8e-acec-0eeae021fcfb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:52:54.353430 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.353405 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-dshm" pod "56e784e5-a048-4f8e-acec-0eeae021fcfb" (UID: "56e784e5-a048-4f8e-acec-0eeae021fcfb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:54.382779 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.382748 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 01:52:54.382942 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.382800 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" podUID="e34d09e7-ec9d-4160-9c74-13b2ccf8d871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 01:52:54.452190 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.452080 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:52:54.452190 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.452127 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e784e5-a048-4f8e-acec-0eeae021fcfb-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:52:54.452190 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.452137 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e784e5-a048-4f8e-acec-0eeae021fcfb-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:52:54.500069 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:54.500037 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:55.059212 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:55.059179 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" Oct 30 01:52:55.059377 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:55.059216 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg" event={"ID":"56e784e5-a048-4f8e-acec-0eeae021fcfb","Type":"ContainerDied","Data":"0543969ce05c54f64b74522946d555b8dc491e8fb6c9d84d6345b439d5b4b15f"} Oct 30 01:52:55.059377 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:55.059263 5998 scope.go:122] "RemoveContainer" containerID="75e072709ae61fe833f5fc2892320bac75235c6067b3be6cfd4cd20bdabc93ac" Oct 30 01:52:55.126123 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:55.123152 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg"] Oct 30 01:52:55.136122 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:55.136070 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/e2e-conflict-a-e338a073-kserve-7c5447cd55-q4wjg"] Oct 30 01:52:55.248668 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:52:55.248634 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:52:55.268719 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:55.268694 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8" Oct 30 01:52:55.361069 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:55.360967 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb8b7b91-c3df-4b25-9eb0-02dad493b607-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8b7b91-c3df-4b25-9eb0-02dad493b607-maas-api-openshift-ca\") pod \"fb8b7b91-c3df-4b25-9eb0-02dad493b607\" (UID: \"fb8b7b91-c3df-4b25-9eb0-02dad493b607\") " Oct 30 01:52:55.361069 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:55.361039 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb8b7b91-c3df-4b25-9eb0-02dad493b607-kube-api-access-lsphv\" (UniqueName: \"kubernetes.io/projected/fb8b7b91-c3df-4b25-9eb0-02dad493b607-kube-api-access-lsphv\") pod \"fb8b7b91-c3df-4b25-9eb0-02dad493b607\" (UID: \"fb8b7b91-c3df-4b25-9eb0-02dad493b607\") " Oct 30 01:52:55.361298 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:55.361192 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb8b7b91-c3df-4b25-9eb0-02dad493b607-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8b7b91-c3df-4b25-9eb0-02dad493b607-maas-api-xks-ca\") pod \"fb8b7b91-c3df-4b25-9eb0-02dad493b607\" (UID: \"fb8b7b91-c3df-4b25-9eb0-02dad493b607\") " Oct 30 01:52:55.361457 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:55.361420 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8b7b91-c3df-4b25-9eb0-02dad493b607-maas-api-openshift-ca" pod "fb8b7b91-c3df-4b25-9eb0-02dad493b607" (UID: "fb8b7b91-c3df-4b25-9eb0-02dad493b607"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:55.361649 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:55.361626 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8b7b91-c3df-4b25-9eb0-02dad493b607-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:52:55.363401 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:55.363372 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8b7b91-c3df-4b25-9eb0-02dad493b607-maas-api-xks-ca" pod "fb8b7b91-c3df-4b25-9eb0-02dad493b607" (UID: "fb8b7b91-c3df-4b25-9eb0-02dad493b607"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:52:55.363489 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:55.363444 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8b7b91-c3df-4b25-9eb0-02dad493b607-kube-api-access-lsphv" pod "fb8b7b91-c3df-4b25-9eb0-02dad493b607" (UID: "fb8b7b91-c3df-4b25-9eb0-02dad493b607"). InnerVolumeSpecName "kube-api-access-lsphv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:55.382779 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:55.382749 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 01:52:55.382894 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:55.382805 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" podUID="e34d09e7-ec9d-4160-9c74-13b2ccf8d871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 01:52:55.462317 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:55.462283 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsphv\" (UniqueName: \"kubernetes.io/projected/fb8b7b91-c3df-4b25-9eb0-02dad493b607-kube-api-access-lsphv\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:52:55.462317 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:55.462312 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8b7b91-c3df-4b25-9eb0-02dad493b607-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.068017 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:56.067986 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8" Oct 30 01:52:56.068017 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:56.068014 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8" event={"ID":"fb8b7b91-c3df-4b25-9eb0-02dad493b607","Type":"ContainerDied","Data":"0d8acb1dccbd1d5426c55f4bc3ca8cab9ed9bc941cb5ff5b6d19167b9bbde3cd"} Oct 30 01:52:56.068336 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:56.068062 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d8acb1dccbd1d5426c55f4bc3ca8cab9ed9bc941cb5ff5b6d19167b9bbde3cd" Oct 30 01:52:56.261438 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:56.261405 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:56.383452 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:56.383357 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 01:52:56.383452 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:56.383422 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" podUID="e34d09e7-ec9d-4160-9c74-13b2ccf8d871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 01:52:56.540831 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:56.540796 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56e784e5-a048-4f8e-acec-0eeae021fcfb" path="/var/lib/kubelet/pods/56e784e5-a048-4f8e-acec-0eeae021fcfb/volumes" Oct 30 01:52:57.382387 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:57.382349 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 01:52:57.382927 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:57.382416 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" podUID="e34d09e7-ec9d-4160-9c74-13b2ccf8d871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 01:52:58.090036 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:58.089999 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:58.382862 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:58.382763 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 01:52:58.382862 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:58.382829 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" podUID="e34d09e7-ec9d-4160-9c74-13b2ccf8d871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 01:52:59.387794 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.387759 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:59.388298 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.388204 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:59.389173 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.389147 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 01:52:59.548316 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.548255 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv"] Oct 30 01:52:59.548990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.548964 5998 cpu_manager.go:414] "containerMap: removing container" podUID="fb8b7b91-c3df-4b25-9eb0-02dad493b607" containerName="revoke-keys" Oct 30 01:52:59.548990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.548987 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb8b7b91-c3df-4b25-9eb0-02dad493b607" containerName="revoke-keys" Oct 30 01:52:59.549242 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.549009 5998 cpu_manager.go:414] "containerMap: removing container" podUID="56e784e5-a048-4f8e-acec-0eeae021fcfb" containerName="main" Oct 30 01:52:59.549242 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.549017 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e784e5-a048-4f8e-acec-0eeae021fcfb" containerName="main" Oct 30 01:52:59.549242 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.549208 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb8b7b91-c3df-4b25-9eb0-02dad493b607" containerName="revoke-keys" Oct 30 01:52:59.549242 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.549227 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e784e5-a048-4f8e-acec-0eeae021fcfb" containerName="main" Oct 30 01:52:59.554544 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.554519 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.573272 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.573231 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-78d5ff\"/\"distinct-w3-78d5ff-kserve-self-signed-certs\"" Oct 30 01:52:59.579003 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.578964 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv"] Oct 30 01:52:59.606425 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.606387 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:59.617353 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.617312 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-model-cache\") pod \"distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.617353 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.617351 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-tls-certs\") pod \"distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.617606 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.617382 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-dshm\") pod \"distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.617606 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.617555 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-home\") pod \"distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.617725 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.617619 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-tmp-dir\") pod \"distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.617792 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.617715 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfbw7\" (UniqueName: \"kubernetes.io/projected/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-kube-api-access-mfbw7\") pod \"distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.720124 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.719170 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-tmp-dir\") pod \"distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.720124 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.719246 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfbw7\" (UniqueName: \"kubernetes.io/projected/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-kube-api-access-mfbw7\") pod \"distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.720124 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.719390 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-model-cache\") pod \"distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.720124 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.719427 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-tls-certs\") pod \"distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.720124 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.719471 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-dshm\") pod \"distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.720124 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.719550 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-home\") pod \"distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.720124 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.720081 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-home\") pod \"distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.720639 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.720401 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-tmp-dir\") pod \"distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.722533 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.721018 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-model-cache\") pod \"distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.740065 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.739546 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-tls-certs\") pod \"distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.742303 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.742277 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-dshm\") pod \"distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.746994 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.746944 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfbw7\" (UniqueName: \"kubernetes.io/projected/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-kube-api-access-mfbw7\") pod \"distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:52:59.855702 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.855662 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:52:59.888351 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:52:59.888314 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:53:00.651475 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:00.651444 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv"] Oct 30 01:53:01.245133 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:01.233204 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" event={"ID":"bc2afe3c-bb27-460f-a7a0-abf3934daf2d","Type":"ContainerStarted","Data":"761fec5f93f95a35af8f4bf8b7fcd8dfa04db9ddbbf7333db5dd63e13eb8765e"} Oct 30 01:53:01.245133 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:01.233251 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" event={"ID":"bc2afe3c-bb27-460f-a7a0-abf3934daf2d","Type":"ContainerStarted","Data":"8c320e1ef556504e3936db0d404d2ce1646f18bb8b39e3c4f1adccf6d5d0ab96"} Oct 30 01:53:01.277926 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:01.277857 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podStartSLOduration=2.27783907 podStartE2EDuration="2.27783907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:01.273306278 +0000 UTC m=+1410.852097361" watchObservedRunningTime="2026-10-30 01:53:01.27783907 +0000 UTC m=+1410.856630153" Oct 30 01:53:01.889648 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:01.889608 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:53:02.155876 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.155792 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww"] Oct 30 01:53:02.159629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.159594 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.182337 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.182296 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:02.189116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.189059 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww"] Oct 30 01:53:02.306290 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.306236 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.306511 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.306395 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.306511 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.306434 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.306609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.306543 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91229273-99e3-4f69-b5d0-ce56b76e309f-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.306609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.306569 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.306709 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.306683 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.306782 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.306767 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw88g\" (UniqueName: \"kubernetes.io/projected/91229273-99e3-4f69-b5d0-ce56b76e309f-kube-api-access-dw88g\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.306840 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.306826 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.306900 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.306862 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.408062 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.407965 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.408283 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.408066 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.408283 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.408127 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.408283 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.408162 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91229273-99e3-4f69-b5d0-ce56b76e309f-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.408283 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.408196 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.408526 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.408476 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.408526 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.408483 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.408627 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.408549 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.408670 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.408621 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw88g\" (UniqueName: \"kubernetes.io/projected/91229273-99e3-4f69-b5d0-ce56b76e309f-kube-api-access-dw88g\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.408724 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.408668 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.408724 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.408718 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.408823 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.408742 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.409032 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.409002 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.409199 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.409181 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91229273-99e3-4f69-b5d0-ce56b76e309f-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.410800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.410775 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.411179 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.411164 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.430991 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.430958 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw88g\" (UniqueName: \"kubernetes.io/projected/91229273-99e3-4f69-b5d0-ce56b76e309f-kube-api-access-dw88g\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.431183 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.430990 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.452068 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.452032 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8"] Oct 30 01:53:02.456924 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.456902 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8" Oct 30 01:53:02.473619 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.473590 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:53:02.473780 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.473739 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:53:02.473847 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.473807 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:02.474075 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.474058 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:53:02.474212 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.474072 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:53:02.477351 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.477326 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8"] Oct 30 01:53:02.610878 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.610835 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmwxz\" (UniqueName: \"kubernetes.io/projected/fdf8bba3-cf02-49be-a66c-ff95056bbc12-kube-api-access-nmwxz\") pod \"maas-api-revoke-sub-0719b37e-74rj8\" (UID: \"fdf8bba3-cf02-49be-a66c-ff95056bbc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8" Oct 30 01:53:02.611072 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.610996 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf8bba3-cf02-49be-a66c-ff95056bbc12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0719b37e-74rj8\" (UID: \"fdf8bba3-cf02-49be-a66c-ff95056bbc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8" Oct 30 01:53:02.611072 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.611029 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf8bba3-cf02-49be-a66c-ff95056bbc12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0719b37e-74rj8\" (UID: \"fdf8bba3-cf02-49be-a66c-ff95056bbc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8" Oct 30 01:53:02.712438 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.712389 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmwxz\" (UniqueName: \"kubernetes.io/projected/fdf8bba3-cf02-49be-a66c-ff95056bbc12-kube-api-access-nmwxz\") pod \"maas-api-revoke-sub-0719b37e-74rj8\" (UID: \"fdf8bba3-cf02-49be-a66c-ff95056bbc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8" Oct 30 01:53:02.712629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.712516 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf8bba3-cf02-49be-a66c-ff95056bbc12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0719b37e-74rj8\" (UID: \"fdf8bba3-cf02-49be-a66c-ff95056bbc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8" Oct 30 01:53:02.712629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.712554 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf8bba3-cf02-49be-a66c-ff95056bbc12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0719b37e-74rj8\" (UID: \"fdf8bba3-cf02-49be-a66c-ff95056bbc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8" Oct 30 01:53:02.713492 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.713466 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf8bba3-cf02-49be-a66c-ff95056bbc12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0719b37e-74rj8\" (UID: \"fdf8bba3-cf02-49be-a66c-ff95056bbc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8" Oct 30 01:53:02.714897 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.714876 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf8bba3-cf02-49be-a66c-ff95056bbc12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0719b37e-74rj8\" (UID: \"fdf8bba3-cf02-49be-a66c-ff95056bbc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8" Oct 30 01:53:02.735591 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.735558 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmwxz\" (UniqueName: \"kubernetes.io/projected/fdf8bba3-cf02-49be-a66c-ff95056bbc12-kube-api-access-nmwxz\") pod \"maas-api-revoke-sub-0719b37e-74rj8\" (UID: \"fdf8bba3-cf02-49be-a66c-ff95056bbc12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8" Oct 30 01:53:02.771343 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.771307 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8" Oct 30 01:53:02.808497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.808451 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww"] Oct 30 01:53:02.812166 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:53:02.812133 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91229273_99e3_4f69_b5d0_ce56b76e309f.slice/crio-bbc52f5c510eea71d02625b3941395f784c1568c1fbb9cd2f7188389f972dba7 WatchSource:0}: Error finding container bbc52f5c510eea71d02625b3941395f784c1568c1fbb9cd2f7188389f972dba7: Status 404 returned error can't find the container with id bbc52f5c510eea71d02625b3941395f784c1568c1fbb9cd2f7188389f972dba7 Oct 30 01:53:02.815007 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.814972 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:53:02.815345 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.815053 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:53:02.815345 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:02.815116 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:53:03.085755 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:03.085728 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8"] Oct 30 01:53:03.088470 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:53:03.088436 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf8bba3_cf02_49be_a66c_ff95056bbc12.slice/crio-4d18f8a76b850b60b9bd57a9b719d738dfd6a08f47a1642238a5b85ffb5b63c3 WatchSource:0}: Error finding container 4d18f8a76b850b60b9bd57a9b719d738dfd6a08f47a1642238a5b85ffb5b63c3: Status 404 returned error can't find the container with id 4d18f8a76b850b60b9bd57a9b719d738dfd6a08f47a1642238a5b85ffb5b63c3 Oct 30 01:53:03.256012 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:03.255974 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8" event={"ID":"fdf8bba3-cf02-49be-a66c-ff95056bbc12","Type":"ContainerStarted","Data":"4d18f8a76b850b60b9bd57a9b719d738dfd6a08f47a1642238a5b85ffb5b63c3"} Oct 30 01:53:03.257733 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:03.257688 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" event={"ID":"91229273-99e3-4f69-b5d0-ce56b76e309f","Type":"ContainerStarted","Data":"629ece0768001b01e69ca3f72de5701a7c2c7a5d553ea8e9ca2712d7e42d85ab"} Oct 30 01:53:03.257886 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:03.257752 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" event={"ID":"91229273-99e3-4f69-b5d0-ce56b76e309f","Type":"ContainerStarted","Data":"bbc52f5c510eea71d02625b3941395f784c1568c1fbb9cd2f7188389f972dba7"} Oct 30 01:53:03.307113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:03.303717 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" podStartSLOduration=1.303697012 podStartE2EDuration="1.303697012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:03.302952475 +0000 UTC m=+1412.881743558" watchObservedRunningTime="2026-10-30 01:53:03.303697012 +0000 UTC m=+1412.882488093" Oct 30 01:53:03.474782 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:03.474744 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:03.476507 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:03.476475 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:53:03.476633 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:03.476533 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" podUID="91229273-99e3-4f69-b5d0-ce56b76e309f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 01:53:03.911173 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:03.905797 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:04.265858 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:04.265759 5998 generic.go:356] "Generic (PLEG): container finished" podID="fdf8bba3-cf02-49be-a66c-ff95056bbc12" containerID="bee4f4a5610b9da77279c70f7836ca4c771a0e575e0a7848886b66e64c7334df" exitCode=0 Oct 30 01:53:04.265858 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:04.265803 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8" event={"ID":"fdf8bba3-cf02-49be-a66c-ff95056bbc12","Type":"ContainerDied","Data":"bee4f4a5610b9da77279c70f7836ca4c771a0e575e0a7848886b66e64c7334df"} Oct 30 01:53:04.475246 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:04.475209 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:53:04.475421 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:04.475274 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" podUID="91229273-99e3-4f69-b5d0-ce56b76e309f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 01:53:05.467684 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.467647 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8" Oct 30 01:53:05.474497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.474473 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:53:05.474604 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.474517 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" podUID="91229273-99e3-4f69-b5d0-ce56b76e309f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 01:53:05.618611 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.618521 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:05.636113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.635158 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w"] Oct 30 01:53:05.636113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.636004 5998 cpu_manager.go:414] "containerMap: removing container" podUID="fdf8bba3-cf02-49be-a66c-ff95056bbc12" containerName="revoke-keys" Oct 30 01:53:05.636113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.636024 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf8bba3-cf02-49be-a66c-ff95056bbc12" containerName="revoke-keys" Oct 30 01:53:05.636436 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.636199 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf8bba3-cf02-49be-a66c-ff95056bbc12" containerName="revoke-keys" Oct 30 01:53:05.640594 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.640546 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w" Oct 30 01:53:05.641111 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.641023 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w"] Oct 30 01:53:05.655307 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.655200 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf8bba3-cf02-49be-a66c-ff95056bbc12-kube-api-access-nmwxz\" (UniqueName: \"kubernetes.io/projected/fdf8bba3-cf02-49be-a66c-ff95056bbc12-kube-api-access-nmwxz\") pod \"fdf8bba3-cf02-49be-a66c-ff95056bbc12\" (UID: \"fdf8bba3-cf02-49be-a66c-ff95056bbc12\") " Oct 30 01:53:05.655456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.655349 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdf8bba3-cf02-49be-a66c-ff95056bbc12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf8bba3-cf02-49be-a66c-ff95056bbc12-maas-api-xks-ca\") pod \"fdf8bba3-cf02-49be-a66c-ff95056bbc12\" (UID: \"fdf8bba3-cf02-49be-a66c-ff95056bbc12\") " Oct 30 01:53:05.655533 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.655504 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf8bba3-cf02-49be-a66c-ff95056bbc12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf8bba3-cf02-49be-a66c-ff95056bbc12-maas-api-openshift-ca\") pod \"fdf8bba3-cf02-49be-a66c-ff95056bbc12\" (UID: \"fdf8bba3-cf02-49be-a66c-ff95056bbc12\") " Oct 30 01:53:05.659831 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.659801 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf8bba3-cf02-49be-a66c-ff95056bbc12-kube-api-access-nmwxz" pod "fdf8bba3-cf02-49be-a66c-ff95056bbc12" (UID: "fdf8bba3-cf02-49be-a66c-ff95056bbc12"). InnerVolumeSpecName "kube-api-access-nmwxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:05.661300 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.661275 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf8bba3-cf02-49be-a66c-ff95056bbc12-maas-api-openshift-ca" pod "fdf8bba3-cf02-49be-a66c-ff95056bbc12" (UID: "fdf8bba3-cf02-49be-a66c-ff95056bbc12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:05.665212 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.664875 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf8bba3-cf02-49be-a66c-ff95056bbc12-maas-api-xks-ca" pod "fdf8bba3-cf02-49be-a66c-ff95056bbc12" (UID: "fdf8bba3-cf02-49be-a66c-ff95056bbc12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:05.757144 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.757104 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2ml9\" (UniqueName: \"kubernetes.io/projected/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-kube-api-access-g2ml9\") pod \"maas-api-revoke-sub-4875c84f-p7z7w\" (UID: \"5318fb38-6f90-4cf4-8d8a-597de8ee1a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w" Oct 30 01:53:05.757331 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.757305 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4875c84f-p7z7w\" (UID: \"5318fb38-6f90-4cf4-8d8a-597de8ee1a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w" Oct 30 01:53:05.757432 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.757414 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4875c84f-p7z7w\" (UID: \"5318fb38-6f90-4cf4-8d8a-597de8ee1a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w" Oct 30 01:53:05.757515 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.757502 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdf8bba3-cf02-49be-a66c-ff95056bbc12-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:05.757567 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.757523 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdf8bba3-cf02-49be-a66c-ff95056bbc12-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:05.757567 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.757538 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmwxz\" (UniqueName: \"kubernetes.io/projected/fdf8bba3-cf02-49be-a66c-ff95056bbc12-kube-api-access-nmwxz\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:05.858655 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.858618 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2ml9\" (UniqueName: \"kubernetes.io/projected/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-kube-api-access-g2ml9\") pod \"maas-api-revoke-sub-4875c84f-p7z7w\" (UID: \"5318fb38-6f90-4cf4-8d8a-597de8ee1a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w" Oct 30 01:53:05.858866 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.858748 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4875c84f-p7z7w\" (UID: \"5318fb38-6f90-4cf4-8d8a-597de8ee1a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w" Oct 30 01:53:05.858866 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.858805 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4875c84f-p7z7w\" (UID: \"5318fb38-6f90-4cf4-8d8a-597de8ee1a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w" Oct 30 01:53:05.859493 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.859465 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4875c84f-p7z7w\" (UID: \"5318fb38-6f90-4cf4-8d8a-597de8ee1a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w" Oct 30 01:53:05.861030 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.861008 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4875c84f-p7z7w\" (UID: \"5318fb38-6f90-4cf4-8d8a-597de8ee1a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w" Oct 30 01:53:05.881172 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.881104 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2ml9\" (UniqueName: \"kubernetes.io/projected/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-kube-api-access-g2ml9\") pod \"maas-api-revoke-sub-4875c84f-p7z7w\" (UID: \"5318fb38-6f90-4cf4-8d8a-597de8ee1a50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w" Oct 30 01:53:05.996870 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:05.996836 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w" Oct 30 01:53:06.094839 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.090821 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb"] Oct 30 01:53:06.098240 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.098128 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.146535 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.146386 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb"] Oct 30 01:53:06.161960 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.161913 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.162218 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.162052 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx6wh\" (UniqueName: \"kubernetes.io/projected/9bcd8c28-6f20-4e7e-b682-729266e83210-kube-api-access-gx6wh\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.162218 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.162102 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.162218 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.162148 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcd8c28-6f20-4e7e-b682-729266e83210-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.162413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.162251 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.162413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.162280 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.162413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.162328 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.162413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.162356 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.162631 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.162533 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.264239 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.264204 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.264427 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.264253 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.264427 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.264312 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.264427 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.264361 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.264604 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.264445 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx6wh\" (UniqueName: \"kubernetes.io/projected/9bcd8c28-6f20-4e7e-b682-729266e83210-kube-api-access-gx6wh\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.264604 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.264471 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.264604 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.264517 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcd8c28-6f20-4e7e-b682-729266e83210-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.264604 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.264558 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.264604 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.264573 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.264934 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.264910 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-credential-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.264934 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.264921 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-workload-certs\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.265255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.265237 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcd8c28-6f20-4e7e-b682-729266e83210-istiod-ca-cert\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.265562 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.265537 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-data\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.265562 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.265551 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-workload-socket\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.266907 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.266891 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-podinfo\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.267102 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.267074 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-envoy\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.281248 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.281217 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8" event={"ID":"fdf8bba3-cf02-49be-a66c-ff95056bbc12","Type":"ContainerDied","Data":"4d18f8a76b850b60b9bd57a9b719d738dfd6a08f47a1642238a5b85ffb5b63c3"} Oct 30 01:53:06.281379 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.281268 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d18f8a76b850b60b9bd57a9b719d738dfd6a08f47a1642238a5b85ffb5b63c3" Oct 30 01:53:06.281379 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.281231 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8" Oct 30 01:53:06.290769 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.290729 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx6wh\" (UniqueName: \"kubernetes.io/projected/9bcd8c28-6f20-4e7e-b682-729266e83210-kube-api-access-gx6wh\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.291231 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.291212 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-token\") pod \"e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.336062 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.336025 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w"] Oct 30 01:53:06.452942 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.452898 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:06.475174 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.475137 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:53:06.475522 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.475190 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" podUID="91229273-99e3-4f69-b5d0-ce56b76e309f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 01:53:06.854496 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.854465 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb"] Oct 30 01:53:06.856998 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:53:06.856969 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bcd8c28_6f20_4e7e_b682_729266e83210.slice/crio-10ec2a7262e78f9fd1e70caee7975f936f6458ac4325fbbee78de810b61d7459 WatchSource:0}: Error finding container 10ec2a7262e78f9fd1e70caee7975f936f6458ac4325fbbee78de810b61d7459: Status 404 returned error can't find the container with id 10ec2a7262e78f9fd1e70caee7975f936f6458ac4325fbbee78de810b61d7459 Oct 30 01:53:06.859527 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.859494 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:53:06.859651 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.859589 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:53:06.859651 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:06.859616 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:53:07.290453 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:07.290413 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w" event={"ID":"5318fb38-6f90-4cf4-8d8a-597de8ee1a50","Type":"ContainerStarted","Data":"f1a6b8fe5b5061ccd65c60e781c0272ba160b4db1ee18bed3e0b22c4b1e1042a"} Oct 30 01:53:07.290663 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:07.290525 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w" event={"ID":"5318fb38-6f90-4cf4-8d8a-597de8ee1a50","Type":"ContainerStarted","Data":"9ab6b883db218b2e0b10e60acce822c8cb8062546ee8ba09dabcedb1a007a583"} Oct 30 01:53:07.292546 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:07.292510 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" event={"ID":"9bcd8c28-6f20-4e7e-b682-729266e83210","Type":"ContainerStarted","Data":"85a9fb6e309aa69dca955b8afc41a231c347ad7046cf26362ff64fa031f96e6a"} Oct 30 01:53:07.292652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:07.292559 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" event={"ID":"9bcd8c28-6f20-4e7e-b682-729266e83210","Type":"ContainerStarted","Data":"10ec2a7262e78f9fd1e70caee7975f936f6458ac4325fbbee78de810b61d7459"} Oct 30 01:53:07.323112 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:07.316547 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:07.351169 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:07.351102 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w" podStartSLOduration=2.351066518 podStartE2EDuration="2.351066518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:07.345482088 +0000 UTC m=+1416.924273172" watchObservedRunningTime="2026-10-30 01:53:07.351066518 +0000 UTC m=+1416.929857602" Oct 30 01:53:07.396020 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:07.395961 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" podStartSLOduration=1.3959458169999999 podStartE2EDuration="1.395945817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:07.392956138 +0000 UTC m=+1416.971747219" watchObservedRunningTime="2026-10-30 01:53:07.395945817 +0000 UTC m=+1416.974736897" Oct 30 01:53:07.456330 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:07.455834 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:07.456330 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:07.455922 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:53:07.456330 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:07.456009 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" podUID="9bcd8c28-6f20-4e7e-b682-729266e83210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 01:53:07.474299 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:07.474268 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:53:07.474499 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:07.474316 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" podUID="91229273-99e3-4f69-b5d0-ce56b76e309f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 01:53:08.306112 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:08.305698 5998 generic.go:356] "Generic (PLEG): container finished" podID="5318fb38-6f90-4cf4-8d8a-597de8ee1a50" containerID="f1a6b8fe5b5061ccd65c60e781c0272ba160b4db1ee18bed3e0b22c4b1e1042a" exitCode=0 Oct 30 01:53:08.310114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:08.307081 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w" event={"ID":"5318fb38-6f90-4cf4-8d8a-597de8ee1a50","Type":"ContainerDied","Data":"f1a6b8fe5b5061ccd65c60e781c0272ba160b4db1ee18bed3e0b22c4b1e1042a"} Oct 30 01:53:08.376353 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:08.376303 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:08.468111 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:08.465973 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:53:08.468111 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:08.466055 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" podUID="9bcd8c28-6f20-4e7e-b682-729266e83210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 01:53:08.474639 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:08.474606 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:53:08.474797 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:08.474659 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" podUID="91229273-99e3-4f69-b5d0-ce56b76e309f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 01:53:09.081824 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:09.081787 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:09.472918 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:09.472208 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:53:09.472918 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:09.472267 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" podUID="9bcd8c28-6f20-4e7e-b682-729266e83210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 01:53:09.474825 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:09.474795 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 01:53:09.474949 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:09.474847 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" podUID="91229273-99e3-4f69-b5d0-ce56b76e309f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 01:53:09.575643 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:09.575613 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w" Oct 30 01:53:09.645181 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:09.637543 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-maas-api-xks-ca\") pod \"5318fb38-6f90-4cf4-8d8a-597de8ee1a50\" (UID: \"5318fb38-6f90-4cf4-8d8a-597de8ee1a50\") " Oct 30 01:53:09.645181 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:09.637660 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-maas-api-openshift-ca\") pod \"5318fb38-6f90-4cf4-8d8a-597de8ee1a50\" (UID: \"5318fb38-6f90-4cf4-8d8a-597de8ee1a50\") " Oct 30 01:53:09.645181 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:09.637779 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-kube-api-access-g2ml9\" (UniqueName: \"kubernetes.io/projected/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-kube-api-access-g2ml9\") pod \"5318fb38-6f90-4cf4-8d8a-597de8ee1a50\" (UID: \"5318fb38-6f90-4cf4-8d8a-597de8ee1a50\") " Oct 30 01:53:09.645181 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:09.638871 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-maas-api-openshift-ca" pod "5318fb38-6f90-4cf4-8d8a-597de8ee1a50" (UID: "5318fb38-6f90-4cf4-8d8a-597de8ee1a50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:09.682212 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:09.673347 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:09.682212 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:09.673958 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-kube-api-access-g2ml9" pod "5318fb38-6f90-4cf4-8d8a-597de8ee1a50" (UID: "5318fb38-6f90-4cf4-8d8a-597de8ee1a50"). InnerVolumeSpecName "kube-api-access-g2ml9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:09.697531 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:09.697208 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-maas-api-xks-ca" pod "5318fb38-6f90-4cf4-8d8a-597de8ee1a50" (UID: "5318fb38-6f90-4cf4-8d8a-597de8ee1a50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:09.762655 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:09.753583 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:09.762655 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:09.753634 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2ml9\" (UniqueName: \"kubernetes.io/projected/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-kube-api-access-g2ml9\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:09.762655 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:09.753650 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5318fb38-6f90-4cf4-8d8a-597de8ee1a50-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:09.894122 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:09.893363 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:53:09.976769 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:09.975665 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:53:10.363737 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:10.362638 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w" Oct 30 01:53:10.363737 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:10.363116 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w" event={"ID":"5318fb38-6f90-4cf4-8d8a-597de8ee1a50","Type":"ContainerDied","Data":"9ab6b883db218b2e0b10e60acce822c8cb8062546ee8ba09dabcedb1a007a583"} Oct 30 01:53:10.363737 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:10.363177 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ab6b883db218b2e0b10e60acce822c8cb8062546ee8ba09dabcedb1a007a583" Oct 30 01:53:10.381960 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:10.381940 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:53:10.454771 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:10.454669 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:53:10.455054 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:10.454737 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" podUID="9bcd8c28-6f20-4e7e-b682-729266e83210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 01:53:10.747642 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:10.747507 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:10.749247 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:10.749145 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:10.751197 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:10.750290 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 01:53:10.760532 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:10.760513 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:11.453792 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:11.453755 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:53:11.453976 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:11.453816 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" podUID="9bcd8c28-6f20-4e7e-b682-729266e83210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 01:53:11.609475 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:11.609436 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:53:12.453746 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:12.453709 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:53:12.454175 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:12.453767 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" podUID="9bcd8c28-6f20-4e7e-b682-729266e83210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 01:53:12.474585 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:12.474548 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:13.460119 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:13.459476 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:53:13.460119 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:13.459536 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" podUID="9bcd8c28-6f20-4e7e-b682-729266e83210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 01:53:14.478115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:14.454655 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:53:14.478115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:14.454716 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" podUID="9bcd8c28-6f20-4e7e-b682-729266e83210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 01:53:14.579333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:14.579283 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:14.681117 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:14.666643 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:14.682555 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:14.675205 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:14.752490 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:14.752379 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:15.004303 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:15.004182 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 01:53:15.274381 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:15.274257 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:53:15.274670 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:15.274636 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" podUID="6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:15.553831 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:15.553725 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 01:53:15.555404 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:15.555374 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:53:15.591322 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:15.591274 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" podUID="9bcd8c28-6f20-4e7e-b682-729266e83210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 01:53:15.592055 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:15.558584 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 01:53:15.638263 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:15.638214 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:15.638888 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:15.638860 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:53:15.747112 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:15.747051 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 01:53:15.747680 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:15.747648 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 01:53:15.855144 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:15.748018 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:53:15.855414 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:15.855394 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:53:15.855578 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:15.765051 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:15.888114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:15.853766 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 01:53:16.006284 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:15.889006 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:53:16.034156 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:16.018622 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 01:53:16.034156 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:16.019014 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:53:16.476228 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:16.467037 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:53:16.476228 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:16.467755 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:53:16.476228 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:16.467805 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" podUID="9bcd8c28-6f20-4e7e-b682-729266e83210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 01:53:16.584870 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:16.584772 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:16.605377 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:16.593676 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:16.723149 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:16.692591 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:53:16.762365 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:16.762254 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:16.865428 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:16.865377 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 01:53:16.865974 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:16.865946 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:16.872187 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:16.872163 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:53:16.886052 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:16.886029 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:53:16.895749 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:16.895724 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:53:16.955032 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:16.955004 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:53:16.957673 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:16.957646 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:53:17.582347 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:17.582311 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:17.582734 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:17.582716 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:17.583650 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:17.583630 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 01:53:17.922765 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:17.922726 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd"] Oct 30 01:53:17.924539 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:17.924513 5998 cpu_manager.go:414] "containerMap: removing container" podUID="5318fb38-6f90-4cf4-8d8a-597de8ee1a50" containerName="revoke-keys" Oct 30 01:53:17.924691 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:17.924677 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5318fb38-6f90-4cf4-8d8a-597de8ee1a50" containerName="revoke-keys" Oct 30 01:53:17.924920 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:17.924907 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="5318fb38-6f90-4cf4-8d8a-597de8ee1a50" containerName="revoke-keys" Oct 30 01:53:17.929986 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:17.929964 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd" Oct 30 01:53:17.945189 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:17.945154 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:53:17.949598 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:17.949566 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd"] Oct 30 01:53:17.950715 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:17.950695 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:53:17.950975 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:17.950941 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:53:17.955260 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:17.955238 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:53:18.092409 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:18.092374 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82965b88-ae62-48bb-896c-6089b43707a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bd2b5f4-68hrd\" (UID: \"82965b88-ae62-48bb-896c-6089b43707a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd" Oct 30 01:53:18.092564 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:18.092419 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79m99\" (UniqueName: \"kubernetes.io/projected/82965b88-ae62-48bb-896c-6089b43707a2-kube-api-access-79m99\") pod \"maas-api-revoke-sub-5bd2b5f4-68hrd\" (UID: \"82965b88-ae62-48bb-896c-6089b43707a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd" Oct 30 01:53:18.092564 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:18.092501 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82965b88-ae62-48bb-896c-6089b43707a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bd2b5f4-68hrd\" (UID: \"82965b88-ae62-48bb-896c-6089b43707a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd" Oct 30 01:53:18.193856 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:18.193751 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82965b88-ae62-48bb-896c-6089b43707a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bd2b5f4-68hrd\" (UID: \"82965b88-ae62-48bb-896c-6089b43707a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd" Oct 30 01:53:18.193856 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:18.193823 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79m99\" (UniqueName: \"kubernetes.io/projected/82965b88-ae62-48bb-896c-6089b43707a2-kube-api-access-79m99\") pod \"maas-api-revoke-sub-5bd2b5f4-68hrd\" (UID: \"82965b88-ae62-48bb-896c-6089b43707a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd" Oct 30 01:53:18.194129 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:18.193883 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82965b88-ae62-48bb-896c-6089b43707a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bd2b5f4-68hrd\" (UID: \"82965b88-ae62-48bb-896c-6089b43707a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd" Oct 30 01:53:18.194789 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:18.194758 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82965b88-ae62-48bb-896c-6089b43707a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bd2b5f4-68hrd\" (UID: \"82965b88-ae62-48bb-896c-6089b43707a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd" Oct 30 01:53:18.196839 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:18.196811 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82965b88-ae62-48bb-896c-6089b43707a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bd2b5f4-68hrd\" (UID: \"82965b88-ae62-48bb-896c-6089b43707a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd" Oct 30 01:53:18.217784 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:18.217751 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79m99\" (UniqueName: \"kubernetes.io/projected/82965b88-ae62-48bb-896c-6089b43707a2-kube-api-access-79m99\") pod \"maas-api-revoke-sub-5bd2b5f4-68hrd\" (UID: \"82965b88-ae62-48bb-896c-6089b43707a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd" Oct 30 01:53:18.252634 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:18.252588 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd" Oct 30 01:53:18.590047 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:18.590019 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd"] Oct 30 01:53:18.595431 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:53:18.594626 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82965b88_ae62_48bb_896c_6089b43707a2.slice/crio-f0db55fb894994483f50a52f369857cdc25b62e39627694a804f9df2fc5fefb7 WatchSource:0}: Error finding container f0db55fb894994483f50a52f369857cdc25b62e39627694a804f9df2fc5fefb7: Status 404 returned error can't find the container with id f0db55fb894994483f50a52f369857cdc25b62e39627694a804f9df2fc5fefb7 Oct 30 01:53:18.797496 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:18.797457 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:18.906186 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:18.905703 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd" event={"ID":"82965b88-ae62-48bb-896c-6089b43707a2","Type":"ContainerStarted","Data":"f0db55fb894994483f50a52f369857cdc25b62e39627694a804f9df2fc5fefb7"} Oct 30 01:53:19.913219 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:19.913184 5998 generic.go:356] "Generic (PLEG): container finished" podID="82965b88-ae62-48bb-896c-6089b43707a2" containerID="8d8bb76c08e7ca811b25dc3cd8ba6ceac8b23643991a296d856623fe6dcc0888" exitCode=0 Oct 30 01:53:19.913628 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:19.913270 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd" event={"ID":"82965b88-ae62-48bb-896c-6089b43707a2","Type":"ContainerDied","Data":"8d8bb76c08e7ca811b25dc3cd8ba6ceac8b23643991a296d856623fe6dcc0888"} Oct 30 01:53:20.408818 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:20.408780 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc"] Oct 30 01:53:20.412735 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:20.412714 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc" Oct 30 01:53:20.434667 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:20.434631 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc"] Oct 30 01:53:20.520266 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:20.520230 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980a30c0-fdfa-4d75-906e-c138dbc9b955-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a20ee2e-rg2lc\" (UID: \"980a30c0-fdfa-4d75-906e-c138dbc9b955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc" Oct 30 01:53:20.520423 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:20.520277 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt66n\" (UniqueName: \"kubernetes.io/projected/980a30c0-fdfa-4d75-906e-c138dbc9b955-kube-api-access-xt66n\") pod \"maas-api-revoke-sub-7a20ee2e-rg2lc\" (UID: \"980a30c0-fdfa-4d75-906e-c138dbc9b955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc" Oct 30 01:53:20.520485 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:20.520375 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980a30c0-fdfa-4d75-906e-c138dbc9b955-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a20ee2e-rg2lc\" (UID: \"980a30c0-fdfa-4d75-906e-c138dbc9b955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc" Oct 30 01:53:20.621488 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:20.621449 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980a30c0-fdfa-4d75-906e-c138dbc9b955-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a20ee2e-rg2lc\" (UID: \"980a30c0-fdfa-4d75-906e-c138dbc9b955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc" Oct 30 01:53:20.621667 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:20.621502 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt66n\" (UniqueName: \"kubernetes.io/projected/980a30c0-fdfa-4d75-906e-c138dbc9b955-kube-api-access-xt66n\") pod \"maas-api-revoke-sub-7a20ee2e-rg2lc\" (UID: \"980a30c0-fdfa-4d75-906e-c138dbc9b955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc" Oct 30 01:53:20.621667 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:20.621527 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980a30c0-fdfa-4d75-906e-c138dbc9b955-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a20ee2e-rg2lc\" (UID: \"980a30c0-fdfa-4d75-906e-c138dbc9b955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc" Oct 30 01:53:20.622414 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:20.622387 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980a30c0-fdfa-4d75-906e-c138dbc9b955-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a20ee2e-rg2lc\" (UID: \"980a30c0-fdfa-4d75-906e-c138dbc9b955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc" Oct 30 01:53:20.623907 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:20.623884 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980a30c0-fdfa-4d75-906e-c138dbc9b955-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a20ee2e-rg2lc\" (UID: \"980a30c0-fdfa-4d75-906e-c138dbc9b955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc" Oct 30 01:53:20.634143 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:20.634106 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:20.644205 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:20.644178 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt66n\" (UniqueName: \"kubernetes.io/projected/980a30c0-fdfa-4d75-906e-c138dbc9b955-kube-api-access-xt66n\") pod \"maas-api-revoke-sub-7a20ee2e-rg2lc\" (UID: \"980a30c0-fdfa-4d75-906e-c138dbc9b955\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc" Oct 30 01:53:20.727580 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:20.727545 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc" Oct 30 01:53:21.041313 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:21.041148 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc"] Oct 30 01:53:21.044801 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:53:21.044761 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980a30c0_fdfa_4d75_906e_c138dbc9b955.slice/crio-193f6ac670a30a72a43f084e4cb5f8570f05673efc503a5f07dac3b4cff6b80c WatchSource:0}: Error finding container 193f6ac670a30a72a43f084e4cb5f8570f05673efc503a5f07dac3b4cff6b80c: Status 404 returned error can't find the container with id 193f6ac670a30a72a43f084e4cb5f8570f05673efc503a5f07dac3b4cff6b80c Oct 30 01:53:21.065493 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:21.065467 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd" Oct 30 01:53:21.228259 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:21.228227 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82965b88-ae62-48bb-896c-6089b43707a2-kube-api-access-79m99\" (UniqueName: \"kubernetes.io/projected/82965b88-ae62-48bb-896c-6089b43707a2-kube-api-access-79m99\") pod \"82965b88-ae62-48bb-896c-6089b43707a2\" (UID: \"82965b88-ae62-48bb-896c-6089b43707a2\") " Oct 30 01:53:21.228382 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:21.228320 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82965b88-ae62-48bb-896c-6089b43707a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82965b88-ae62-48bb-896c-6089b43707a2-maas-api-xks-ca\") pod \"82965b88-ae62-48bb-896c-6089b43707a2\" (UID: \"82965b88-ae62-48bb-896c-6089b43707a2\") " Oct 30 01:53:21.228466 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:21.228453 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82965b88-ae62-48bb-896c-6089b43707a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82965b88-ae62-48bb-896c-6089b43707a2-maas-api-openshift-ca\") pod \"82965b88-ae62-48bb-896c-6089b43707a2\" (UID: \"82965b88-ae62-48bb-896c-6089b43707a2\") " Oct 30 01:53:21.228858 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:21.228829 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82965b88-ae62-48bb-896c-6089b43707a2-maas-api-openshift-ca" pod "82965b88-ae62-48bb-896c-6089b43707a2" (UID: "82965b88-ae62-48bb-896c-6089b43707a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:21.230442 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:21.230417 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82965b88-ae62-48bb-896c-6089b43707a2-kube-api-access-79m99" pod "82965b88-ae62-48bb-896c-6089b43707a2" (UID: "82965b88-ae62-48bb-896c-6089b43707a2"). InnerVolumeSpecName "kube-api-access-79m99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:21.230525 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:21.230500 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82965b88-ae62-48bb-896c-6089b43707a2-maas-api-xks-ca" pod "82965b88-ae62-48bb-896c-6089b43707a2" (UID: "82965b88-ae62-48bb-896c-6089b43707a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:21.330262 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:21.330234 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82965b88-ae62-48bb-896c-6089b43707a2-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:21.330262 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:21.330260 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79m99\" (UniqueName: \"kubernetes.io/projected/82965b88-ae62-48bb-896c-6089b43707a2-kube-api-access-79m99\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:21.330425 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:21.330270 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82965b88-ae62-48bb-896c-6089b43707a2-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:21.930896 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:21.930861 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd" Oct 30 01:53:21.930896 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:21.930884 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd" event={"ID":"82965b88-ae62-48bb-896c-6089b43707a2","Type":"ContainerDied","Data":"f0db55fb894994483f50a52f369857cdc25b62e39627694a804f9df2fc5fefb7"} Oct 30 01:53:21.931224 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:21.930938 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0db55fb894994483f50a52f369857cdc25b62e39627694a804f9df2fc5fefb7" Oct 30 01:53:21.932927 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:21.932891 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc" event={"ID":"980a30c0-fdfa-4d75-906e-c138dbc9b955","Type":"ContainerStarted","Data":"c7d7783dfe04e555140bc714a3053fe840e9b5757db5aeeb3061082b695e96cf"} Oct 30 01:53:21.933045 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:21.932942 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc" event={"ID":"980a30c0-fdfa-4d75-906e-c138dbc9b955","Type":"ContainerStarted","Data":"193f6ac670a30a72a43f084e4cb5f8570f05673efc503a5f07dac3b4cff6b80c"} Oct 30 01:53:21.973009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:21.972947 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc" podStartSLOduration=1.97293449 podStartE2EDuration="1.97293449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:21.971605361 +0000 UTC m=+1431.550396443" watchObservedRunningTime="2026-10-30 01:53:21.97293449 +0000 UTC m=+1431.551725571" Oct 30 01:53:22.403458 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:22.403426 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:22.941680 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:22.941603 5998 generic.go:356] "Generic (PLEG): container finished" podID="980a30c0-fdfa-4d75-906e-c138dbc9b955" containerID="c7d7783dfe04e555140bc714a3053fe840e9b5757db5aeeb3061082b695e96cf" exitCode=0 Oct 30 01:53:22.941896 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:22.941700 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc" event={"ID":"980a30c0-fdfa-4d75-906e-c138dbc9b955","Type":"ContainerDied","Data":"c7d7783dfe04e555140bc714a3053fe840e9b5757db5aeeb3061082b695e96cf"} Oct 30 01:53:24.066728 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:24.066702 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:24.090441 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:24.090417 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc" Oct 30 01:53:24.165680 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:24.165638 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/980a30c0-fdfa-4d75-906e-c138dbc9b955-kube-api-access-xt66n\" (UniqueName: \"kubernetes.io/projected/980a30c0-fdfa-4d75-906e-c138dbc9b955-kube-api-access-xt66n\") pod \"980a30c0-fdfa-4d75-906e-c138dbc9b955\" (UID: \"980a30c0-fdfa-4d75-906e-c138dbc9b955\") " Oct 30 01:53:24.165855 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:24.165740 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/980a30c0-fdfa-4d75-906e-c138dbc9b955-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980a30c0-fdfa-4d75-906e-c138dbc9b955-maas-api-xks-ca\") pod \"980a30c0-fdfa-4d75-906e-c138dbc9b955\" (UID: \"980a30c0-fdfa-4d75-906e-c138dbc9b955\") " Oct 30 01:53:24.165855 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:24.165820 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/980a30c0-fdfa-4d75-906e-c138dbc9b955-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980a30c0-fdfa-4d75-906e-c138dbc9b955-maas-api-openshift-ca\") pod \"980a30c0-fdfa-4d75-906e-c138dbc9b955\" (UID: \"980a30c0-fdfa-4d75-906e-c138dbc9b955\") " Oct 30 01:53:24.167231 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:24.166877 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980a30c0-fdfa-4d75-906e-c138dbc9b955-maas-api-openshift-ca" pod "980a30c0-fdfa-4d75-906e-c138dbc9b955" (UID: "980a30c0-fdfa-4d75-906e-c138dbc9b955"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:24.171508 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:24.171454 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980a30c0-fdfa-4d75-906e-c138dbc9b955-kube-api-access-xt66n" pod "980a30c0-fdfa-4d75-906e-c138dbc9b955" (UID: "980a30c0-fdfa-4d75-906e-c138dbc9b955"). InnerVolumeSpecName "kube-api-access-xt66n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:24.177903 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:24.177867 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/980a30c0-fdfa-4d75-906e-c138dbc9b955-maas-api-xks-ca" pod "980a30c0-fdfa-4d75-906e-c138dbc9b955" (UID: "980a30c0-fdfa-4d75-906e-c138dbc9b955"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:24.268230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:24.268140 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980a30c0-fdfa-4d75-906e-c138dbc9b955-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:24.268230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:24.268175 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt66n\" (UniqueName: \"kubernetes.io/projected/980a30c0-fdfa-4d75-906e-c138dbc9b955-kube-api-access-xt66n\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:24.268230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:24.268186 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980a30c0-fdfa-4d75-906e-c138dbc9b955-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:24.957054 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:24.957017 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc" Oct 30 01:53:24.957222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:24.957056 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc" event={"ID":"980a30c0-fdfa-4d75-906e-c138dbc9b955","Type":"ContainerDied","Data":"193f6ac670a30a72a43f084e4cb5f8570f05673efc503a5f07dac3b4cff6b80c"} Oct 30 01:53:24.957222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:24.957125 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="193f6ac670a30a72a43f084e4cb5f8570f05673efc503a5f07dac3b4cff6b80c" Oct 30 01:53:25.217714 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:53:25.217616 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:53:25.453673 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:25.453635 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:25.625124 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:25.624976 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:27.265488 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:27.265452 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:28.787477 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:28.787437 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:29.927041 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:29.927001 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb"] Oct 30 01:53:29.928403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:29.928376 5998 cpu_manager.go:414] "containerMap: removing container" podUID="980a30c0-fdfa-4d75-906e-c138dbc9b955" containerName="revoke-keys" Oct 30 01:53:29.928564 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:29.928548 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="980a30c0-fdfa-4d75-906e-c138dbc9b955" containerName="revoke-keys" Oct 30 01:53:29.928675 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:29.928662 5998 cpu_manager.go:414] "containerMap: removing container" podUID="82965b88-ae62-48bb-896c-6089b43707a2" containerName="revoke-keys" Oct 30 01:53:29.928776 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:29.928763 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82965b88-ae62-48bb-896c-6089b43707a2" containerName="revoke-keys" Oct 30 01:53:29.929012 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:29.928996 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="980a30c0-fdfa-4d75-906e-c138dbc9b955" containerName="revoke-keys" Oct 30 01:53:29.929136 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:29.929124 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="82965b88-ae62-48bb-896c-6089b43707a2" containerName="revoke-keys" Oct 30 01:53:29.944575 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:29.944548 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb" Oct 30 01:53:29.975353 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:29.975320 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb"] Oct 30 01:53:30.004392 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.004359 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85cfaa98-fbb1-46c9-800e-1a5a190abd15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24cf977b-sgfnb\" (UID: \"85cfaa98-fbb1-46c9-800e-1a5a190abd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb" Oct 30 01:53:30.004703 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.004678 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85cfaa98-fbb1-46c9-800e-1a5a190abd15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24cf977b-sgfnb\" (UID: \"85cfaa98-fbb1-46c9-800e-1a5a190abd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb" Oct 30 01:53:30.004868 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.004851 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prsd2\" (UniqueName: \"kubernetes.io/projected/85cfaa98-fbb1-46c9-800e-1a5a190abd15-kube-api-access-prsd2\") pod \"maas-api-revoke-sub-24cf977b-sgfnb\" (UID: \"85cfaa98-fbb1-46c9-800e-1a5a190abd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb" Oct 30 01:53:30.047117 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.019478 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:53:30.047117 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.019555 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:53:30.047117 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.019601 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:53:30.047117 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.019679 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:53:30.114136 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.112876 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85cfaa98-fbb1-46c9-800e-1a5a190abd15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24cf977b-sgfnb\" (UID: \"85cfaa98-fbb1-46c9-800e-1a5a190abd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb" Oct 30 01:53:30.114136 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.113057 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85cfaa98-fbb1-46c9-800e-1a5a190abd15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24cf977b-sgfnb\" (UID: \"85cfaa98-fbb1-46c9-800e-1a5a190abd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb" Oct 30 01:53:30.114136 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.113143 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prsd2\" (UniqueName: \"kubernetes.io/projected/85cfaa98-fbb1-46c9-800e-1a5a190abd15-kube-api-access-prsd2\") pod \"maas-api-revoke-sub-24cf977b-sgfnb\" (UID: \"85cfaa98-fbb1-46c9-800e-1a5a190abd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb" Oct 30 01:53:30.115718 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.115687 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85cfaa98-fbb1-46c9-800e-1a5a190abd15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24cf977b-sgfnb\" (UID: \"85cfaa98-fbb1-46c9-800e-1a5a190abd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb" Oct 30 01:53:30.265417 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.187408 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85cfaa98-fbb1-46c9-800e-1a5a190abd15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24cf977b-sgfnb\" (UID: \"85cfaa98-fbb1-46c9-800e-1a5a190abd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb" Oct 30 01:53:30.265417 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.252735 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prsd2\" (UniqueName: \"kubernetes.io/projected/85cfaa98-fbb1-46c9-800e-1a5a190abd15-kube-api-access-prsd2\") pod \"maas-api-revoke-sub-24cf977b-sgfnb\" (UID: \"85cfaa98-fbb1-46c9-800e-1a5a190abd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb" Oct 30 01:53:30.578184 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.576405 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:53:30.580832 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.580806 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb" Oct 30 01:53:30.872248 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.872203 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:30.883452 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.883413 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:30.893683 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.893643 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:30.894279 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.893915 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:30.905213 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.905171 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:30.980055 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:30.980009 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:31.375577 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:31.375533 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:31.492029 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:31.491979 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 01:53:31.853399 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:31.852188 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:31.984661 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:31.960494 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:31.984661 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:31.961238 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:53:31.984661 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:31.967400 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb"] Oct 30 01:53:31.984661 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:31.969947 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:31.984661 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:31.970282 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:53:31.984661 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:31.971389 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 01:53:31.984661 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:31.971645 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:53:31.984661 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:31.983168 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:31.984661 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:31.983547 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:53:32.040130 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:32.022765 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:53:32.061534 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:32.057955 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:53:32.411079 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:32.397943 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:32.411079 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:32.398786 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:53:32.517130 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:32.504552 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:32.517130 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:32.504930 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:53:32.807790 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:32.806875 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:53:32.807790 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:32.806933 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:53:32.819234 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:32.812447 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb" event={"ID":"85cfaa98-fbb1-46c9-800e-1a5a190abd15","Type":"ContainerStarted","Data":"9c2d15b1d7e02b080bdba89d011da2d74ef6760e56ad1e33410e92b127fe42ad"} Oct 30 01:53:32.832242 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:32.832209 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:53:32.832500 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:32.832483 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:53:33.686339 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:33.686199 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:33.821156 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:33.821072 5998 generic.go:356] "Generic (PLEG): container finished" podID="85cfaa98-fbb1-46c9-800e-1a5a190abd15" containerID="45fd57e9751848a3a193bd3afaf1263d706cdf57d0fa9966aa9e7be2f12110cb" exitCode=0 Oct 30 01:53:33.821289 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:33.821124 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb" event={"ID":"85cfaa98-fbb1-46c9-800e-1a5a190abd15","Type":"ContainerDied","Data":"45fd57e9751848a3a193bd3afaf1263d706cdf57d0fa9966aa9e7be2f12110cb"} Oct 30 01:53:34.010789 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:34.010714 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s"] Oct 30 01:53:34.014614 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:34.014588 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s" Oct 30 01:53:34.040161 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:34.040130 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s"] Oct 30 01:53:34.146416 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:34.146379 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1c517a-5c98-447e-b804-75933d6f64bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecbf2724-fbv6s\" (UID: \"8d1c517a-5c98-447e-b804-75933d6f64bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s" Oct 30 01:53:34.146416 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:34.146422 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1c517a-5c98-447e-b804-75933d6f64bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecbf2724-fbv6s\" (UID: \"8d1c517a-5c98-447e-b804-75933d6f64bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s" Oct 30 01:53:34.146696 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:34.146570 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcp75\" (UniqueName: \"kubernetes.io/projected/8d1c517a-5c98-447e-b804-75933d6f64bf-kube-api-access-rcp75\") pod \"maas-api-revoke-sub-ecbf2724-fbv6s\" (UID: \"8d1c517a-5c98-447e-b804-75933d6f64bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s" Oct 30 01:53:34.248447 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:34.248405 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1c517a-5c98-447e-b804-75933d6f64bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecbf2724-fbv6s\" (UID: \"8d1c517a-5c98-447e-b804-75933d6f64bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s" Oct 30 01:53:34.248657 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:34.248463 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1c517a-5c98-447e-b804-75933d6f64bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecbf2724-fbv6s\" (UID: \"8d1c517a-5c98-447e-b804-75933d6f64bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s" Oct 30 01:53:34.248657 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:34.248618 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcp75\" (UniqueName: \"kubernetes.io/projected/8d1c517a-5c98-447e-b804-75933d6f64bf-kube-api-access-rcp75\") pod \"maas-api-revoke-sub-ecbf2724-fbv6s\" (UID: \"8d1c517a-5c98-447e-b804-75933d6f64bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s" Oct 30 01:53:34.249246 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:34.249219 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1c517a-5c98-447e-b804-75933d6f64bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecbf2724-fbv6s\" (UID: \"8d1c517a-5c98-447e-b804-75933d6f64bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s" Oct 30 01:53:34.251006 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:34.250980 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1c517a-5c98-447e-b804-75933d6f64bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecbf2724-fbv6s\" (UID: \"8d1c517a-5c98-447e-b804-75933d6f64bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s" Oct 30 01:53:34.272163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:34.272075 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcp75\" (UniqueName: \"kubernetes.io/projected/8d1c517a-5c98-447e-b804-75933d6f64bf-kube-api-access-rcp75\") pod \"maas-api-revoke-sub-ecbf2724-fbv6s\" (UID: \"8d1c517a-5c98-447e-b804-75933d6f64bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s" Oct 30 01:53:34.329178 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:34.329145 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s" Oct 30 01:53:34.352569 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:34.352536 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:34.639757 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:34.639732 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s"] Oct 30 01:53:34.642530 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:53:34.642496 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1c517a_5c98_447e_b804_75933d6f64bf.slice/crio-bfe6af95006d20a5ae0aad74c0d3134995a5ad3335aa817cbabc9b857d0d3fe6 WatchSource:0}: Error finding container bfe6af95006d20a5ae0aad74c0d3134995a5ad3335aa817cbabc9b857d0d3fe6: Status 404 returned error can't find the container with id bfe6af95006d20a5ae0aad74c0d3134995a5ad3335aa817cbabc9b857d0d3fe6 Oct 30 01:53:34.830387 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:34.830348 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s" event={"ID":"8d1c517a-5c98-447e-b804-75933d6f64bf","Type":"ContainerStarted","Data":"bfe6af95006d20a5ae0aad74c0d3134995a5ad3335aa817cbabc9b857d0d3fe6"} Oct 30 01:53:34.974832 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:34.974800 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb" Oct 30 01:53:35.057800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:35.057765 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85cfaa98-fbb1-46c9-800e-1a5a190abd15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85cfaa98-fbb1-46c9-800e-1a5a190abd15-maas-api-openshift-ca\") pod \"85cfaa98-fbb1-46c9-800e-1a5a190abd15\" (UID: \"85cfaa98-fbb1-46c9-800e-1a5a190abd15\") " Oct 30 01:53:35.057971 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:35.057867 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85cfaa98-fbb1-46c9-800e-1a5a190abd15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85cfaa98-fbb1-46c9-800e-1a5a190abd15-maas-api-xks-ca\") pod \"85cfaa98-fbb1-46c9-800e-1a5a190abd15\" (UID: \"85cfaa98-fbb1-46c9-800e-1a5a190abd15\") " Oct 30 01:53:35.058055 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:35.058037 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85cfaa98-fbb1-46c9-800e-1a5a190abd15-kube-api-access-prsd2\" (UniqueName: \"kubernetes.io/projected/85cfaa98-fbb1-46c9-800e-1a5a190abd15-kube-api-access-prsd2\") pod \"85cfaa98-fbb1-46c9-800e-1a5a190abd15\" (UID: \"85cfaa98-fbb1-46c9-800e-1a5a190abd15\") " Oct 30 01:53:35.058184 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:35.058158 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85cfaa98-fbb1-46c9-800e-1a5a190abd15-maas-api-openshift-ca" pod "85cfaa98-fbb1-46c9-800e-1a5a190abd15" (UID: "85cfaa98-fbb1-46c9-800e-1a5a190abd15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:35.058539 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:35.058503 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85cfaa98-fbb1-46c9-800e-1a5a190abd15-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:35.060130 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:35.060081 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85cfaa98-fbb1-46c9-800e-1a5a190abd15-maas-api-xks-ca" pod "85cfaa98-fbb1-46c9-800e-1a5a190abd15" (UID: "85cfaa98-fbb1-46c9-800e-1a5a190abd15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:35.060335 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:35.060313 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85cfaa98-fbb1-46c9-800e-1a5a190abd15-kube-api-access-prsd2" pod "85cfaa98-fbb1-46c9-800e-1a5a190abd15" (UID: "85cfaa98-fbb1-46c9-800e-1a5a190abd15"). InnerVolumeSpecName "kube-api-access-prsd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:35.159297 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:35.159266 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prsd2\" (UniqueName: \"kubernetes.io/projected/85cfaa98-fbb1-46c9-800e-1a5a190abd15-kube-api-access-prsd2\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:35.159297 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:35.159296 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85cfaa98-fbb1-46c9-800e-1a5a190abd15-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:35.282683 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:35.282593 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:35.841621 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:35.841581 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb" event={"ID":"85cfaa98-fbb1-46c9-800e-1a5a190abd15","Type":"ContainerDied","Data":"9c2d15b1d7e02b080bdba89d011da2d74ef6760e56ad1e33410e92b127fe42ad"} Oct 30 01:53:35.842109 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:35.841673 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c2d15b1d7e02b080bdba89d011da2d74ef6760e56ad1e33410e92b127fe42ad" Oct 30 01:53:35.842430 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:35.842407 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb" Oct 30 01:53:35.843871 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:35.843846 5998 generic.go:356] "Generic (PLEG): container finished" podID="8d1c517a-5c98-447e-b804-75933d6f64bf" containerID="27edfbb90be82e7753a3ee20167cf11843ff72ecb768d4a93ec914fdc8e5ee12" exitCode=0 Oct 30 01:53:35.844003 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:35.843942 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s" event={"ID":"8d1c517a-5c98-447e-b804-75933d6f64bf","Type":"ContainerDied","Data":"27edfbb90be82e7753a3ee20167cf11843ff72ecb768d4a93ec914fdc8e5ee12"} Oct 30 01:53:36.984848 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:36.984711 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:37.034287 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:37.034244 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s" Oct 30 01:53:37.087289 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:37.087254 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1c517a-5c98-447e-b804-75933d6f64bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1c517a-5c98-447e-b804-75933d6f64bf-maas-api-openshift-ca\") pod \"8d1c517a-5c98-447e-b804-75933d6f64bf\" (UID: \"8d1c517a-5c98-447e-b804-75933d6f64bf\") " Oct 30 01:53:37.087473 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:37.087411 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1c517a-5c98-447e-b804-75933d6f64bf-kube-api-access-rcp75\" (UniqueName: \"kubernetes.io/projected/8d1c517a-5c98-447e-b804-75933d6f64bf-kube-api-access-rcp75\") pod \"8d1c517a-5c98-447e-b804-75933d6f64bf\" (UID: \"8d1c517a-5c98-447e-b804-75933d6f64bf\") " Oct 30 01:53:37.087473 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:37.087451 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d1c517a-5c98-447e-b804-75933d6f64bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1c517a-5c98-447e-b804-75933d6f64bf-maas-api-xks-ca\") pod \"8d1c517a-5c98-447e-b804-75933d6f64bf\" (UID: \"8d1c517a-5c98-447e-b804-75933d6f64bf\") " Oct 30 01:53:37.087663 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:37.087639 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1c517a-5c98-447e-b804-75933d6f64bf-maas-api-openshift-ca" pod "8d1c517a-5c98-447e-b804-75933d6f64bf" (UID: "8d1c517a-5c98-447e-b804-75933d6f64bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:37.087925 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:37.087907 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1c517a-5c98-447e-b804-75933d6f64bf-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:37.089666 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:37.089630 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d1c517a-5c98-447e-b804-75933d6f64bf-maas-api-xks-ca" pod "8d1c517a-5c98-447e-b804-75933d6f64bf" (UID: "8d1c517a-5c98-447e-b804-75933d6f64bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:37.089843 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:37.089821 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1c517a-5c98-447e-b804-75933d6f64bf-kube-api-access-rcp75" pod "8d1c517a-5c98-447e-b804-75933d6f64bf" (UID: "8d1c517a-5c98-447e-b804-75933d6f64bf"). InnerVolumeSpecName "kube-api-access-rcp75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:37.189456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:37.189419 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcp75\" (UniqueName: \"kubernetes.io/projected/8d1c517a-5c98-447e-b804-75933d6f64bf-kube-api-access-rcp75\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:37.189456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:37.189451 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1c517a-5c98-447e-b804-75933d6f64bf-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:37.865661 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:37.865624 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s" event={"ID":"8d1c517a-5c98-447e-b804-75933d6f64bf","Type":"ContainerDied","Data":"bfe6af95006d20a5ae0aad74c0d3134995a5ad3335aa817cbabc9b857d0d3fe6"} Oct 30 01:53:37.865825 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:37.865672 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfe6af95006d20a5ae0aad74c0d3134995a5ad3335aa817cbabc9b857d0d3fe6" Oct 30 01:53:37.865825 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:37.865637 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s" Oct 30 01:53:38.516359 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:38.516310 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:39.538533 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.538493 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw"] Oct 30 01:53:39.539246 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.539220 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8d1c517a-5c98-447e-b804-75933d6f64bf" containerName="revoke-keys" Oct 30 01:53:39.539246 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.539244 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d1c517a-5c98-447e-b804-75933d6f64bf" containerName="revoke-keys" Oct 30 01:53:39.539435 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.539285 5998 cpu_manager.go:414] "containerMap: removing container" podUID="85cfaa98-fbb1-46c9-800e-1a5a190abd15" containerName="revoke-keys" Oct 30 01:53:39.539435 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.539292 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85cfaa98-fbb1-46c9-800e-1a5a190abd15" containerName="revoke-keys" Oct 30 01:53:39.539435 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.539400 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="85cfaa98-fbb1-46c9-800e-1a5a190abd15" containerName="revoke-keys" Oct 30 01:53:39.539435 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.539421 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d1c517a-5c98-447e-b804-75933d6f64bf" containerName="revoke-keys" Oct 30 01:53:39.543480 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.543455 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.572587 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.572557 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-78d5ff\"/\"distinct-2-w3-78d5ff-kserve-self-signed-certs\"" Oct 30 01:53:39.577466 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.577439 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw"] Oct 30 01:53:39.608946 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.608915 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:39.620358 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.620324 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-tmp-dir\") pod \"distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.620545 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.620395 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-model-cache\") pod \"distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.620545 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.620448 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e218532-053c-41f2-8114-ab1385bbc759-tls-certs\") pod \"distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.620545 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.620494 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-home\") pod \"distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.620682 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.620648 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-dshm\") pod \"distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.620733 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.620684 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9ttz\" (UniqueName: \"kubernetes.io/projected/4e218532-053c-41f2-8114-ab1385bbc759-kube-api-access-b9ttz\") pod \"distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.721800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.721759 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-tmp-dir\") pod \"distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.722002 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.721830 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-model-cache\") pod \"distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.722002 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.721882 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e218532-053c-41f2-8114-ab1385bbc759-tls-certs\") pod \"distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.722002 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.721933 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-home\") pod \"distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.722002 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.721981 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-dshm\") pod \"distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.722264 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.722005 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9ttz\" (UniqueName: \"kubernetes.io/projected/4e218532-053c-41f2-8114-ab1385bbc759-kube-api-access-b9ttz\") pod \"distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.722829 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.722794 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-model-cache\") pod \"distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.723259 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.723233 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-home\") pod \"distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.723623 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.723598 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-tmp-dir\") pod \"distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.725378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.725349 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e218532-053c-41f2-8114-ab1385bbc759-tls-certs\") pod \"distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.725590 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.725572 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-dshm\") pod \"distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.766754 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.766681 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9ttz\" (UniqueName: \"kubernetes.io/projected/4e218532-053c-41f2-8114-ab1385bbc759-kube-api-access-b9ttz\") pod \"distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:39.860169 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:39.859645 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:40.106939 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.106890 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw"] Oct 30 01:53:40.112759 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.112685 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.143076 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.142934 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw"] Oct 30 01:53:40.201704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.201672 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw"] Oct 30 01:53:40.206329 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:53:40.205613 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e218532_053c_41f2_8114_ab1385bbc759.slice/crio-28099f2ba918810f53aba9b21a8439a976e417ff5e932bde6e586e085c9ef6d6 WatchSource:0}: Error finding container 28099f2ba918810f53aba9b21a8439a976e417ff5e932bde6e586e085c9ef6d6: Status 404 returned error can't find the container with id 28099f2ba918810f53aba9b21a8439a976e417ff5e932bde6e586e085c9ef6d6 Oct 30 01:53:40.230906 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.230873 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.233193 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.230927 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.233193 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.230986 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.233193 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.231016 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.233193 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.231068 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2l2z\" (UniqueName: \"kubernetes.io/projected/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-kube-api-access-m2l2z\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.233193 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.231151 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.233193 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.231247 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.233193 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.231280 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.233193 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.231306 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.300127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.300061 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:40.333308 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.333267 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.333484 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.333327 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.333484 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.333391 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.334579 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.333766 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.334579 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.333789 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-data\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.334579 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.333851 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2l2z\" (UniqueName: \"kubernetes.io/projected/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-kube-api-access-m2l2z\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.334579 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.333903 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.334579 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.333963 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.334579 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.333990 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.334579 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.334007 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.334579 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.334059 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-workload-certs\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.334579 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.334336 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-credential-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.335229 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.334938 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istiod-ca-cert\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.335229 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.335055 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-workload-socket\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.338468 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.338271 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-envoy\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.338957 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.338928 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-podinfo\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.357941 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.357912 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-token\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.359267 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.359247 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2l2z\" (UniqueName: \"kubernetes.io/projected/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-kube-api-access-m2l2z\") pod \"e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.433232 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.433193 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:40.767534 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.767499 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw"] Oct 30 01:53:40.770607 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:53:40.770576 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c8b61a_b6aa_4f36_86c5_bd5cb6532ff5.slice/crio-891564830b21c6551cf88f67988a4d2a1ab047fee4d8071a3fd737dddc79c50d WatchSource:0}: Error finding container 891564830b21c6551cf88f67988a4d2a1ab047fee4d8071a3fd737dddc79c50d: Status 404 returned error can't find the container with id 891564830b21c6551cf88f67988a4d2a1ab047fee4d8071a3fd737dddc79c50d Oct 30 01:53:40.780373 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.780334 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:53:40.780473 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.780408 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:53:40.780473 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.780441 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:53:40.897181 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.897130 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" event={"ID":"4e218532-053c-41f2-8114-ab1385bbc759","Type":"ContainerStarted","Data":"831fde1f0a08333d72fc773b58fc3c3cd379a0b256722e9b5f7d93237d8cab19"} Oct 30 01:53:40.897416 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.897204 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" event={"ID":"4e218532-053c-41f2-8114-ab1385bbc759","Type":"ContainerStarted","Data":"28099f2ba918810f53aba9b21a8439a976e417ff5e932bde6e586e085c9ef6d6"} Oct 30 01:53:40.899848 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.899813 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" event={"ID":"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5","Type":"ContainerStarted","Data":"c9945b7fa8fd6570ff58163afae1392dcc251e12fa5c4a4ef0beeda9fe9c65d8"} Oct 30 01:53:40.899979 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.899891 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" event={"ID":"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5","Type":"ContainerStarted","Data":"891564830b21c6551cf88f67988a4d2a1ab047fee4d8071a3fd737dddc79c50d"} Oct 30 01:53:40.946398 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.946340 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podStartSLOduration=1.946323156 podStartE2EDuration="1.946323156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:40.9422737 +0000 UTC m=+1450.521064782" watchObservedRunningTime="2026-10-30 01:53:40.946323156 +0000 UTC m=+1450.525114236" Oct 30 01:53:40.970103 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.970053 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx"] Oct 30 01:53:40.974577 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.974554 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:40.989668 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:40.989601 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" podStartSLOduration=0.98958203 podStartE2EDuration="989.58203ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:40.987934802 +0000 UTC m=+1450.566725883" watchObservedRunningTime="2026-10-30 01:53:40.98958203 +0000 UTC m=+1450.568373112" Oct 30 01:53:41.000112 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.000054 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx"] Oct 30 01:53:41.045957 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.045846 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff075586-a6a8-4eed-819d-19c55abfa057-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.045957 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.045941 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlb9w\" (UniqueName: \"kubernetes.io/projected/ff075586-a6a8-4eed-819d-19c55abfa057-kube-api-access-vlb9w\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.046232 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.045979 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.046232 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.046019 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.046232 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.046144 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff075586-a6a8-4eed-819d-19c55abfa057-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.046232 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.046192 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.046421 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.046336 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff075586-a6a8-4eed-819d-19c55abfa057-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.046471 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.046420 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.046522 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.046494 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.126117 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.125684 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:53:41.148157 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.148111 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff075586-a6a8-4eed-819d-19c55abfa057-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.148356 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.148193 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlb9w\" (UniqueName: \"kubernetes.io/projected/ff075586-a6a8-4eed-819d-19c55abfa057-kube-api-access-vlb9w\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.148356 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.148250 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.148356 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.148289 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.148539 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.148376 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff075586-a6a8-4eed-819d-19c55abfa057-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.148539 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.148416 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.148539 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.148523 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff075586-a6a8-4eed-819d-19c55abfa057-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.148704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.148548 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.148704 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.148581 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.149148 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.149120 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.150149 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.149796 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff075586-a6a8-4eed-819d-19c55abfa057-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.151755 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.151730 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.154917 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.152524 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.154917 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.152866 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.154917 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.152978 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.155631 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.155593 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff075586-a6a8-4eed-819d-19c55abfa057-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.175371 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.175336 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlb9w\" (UniqueName: \"kubernetes.io/projected/ff075586-a6a8-4eed-819d-19c55abfa057-kube-api-access-vlb9w\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.175988 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.175956 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff075586-a6a8-4eed-819d-19c55abfa057-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.309957 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.309851 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:41.434126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.434001 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:41.443303 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.443265 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:53:41.443487 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.443332 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" podUID="d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 01:53:41.642717 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.642624 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx"] Oct 30 01:53:41.655591 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.655557 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:53:41.655696 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.655625 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:53:41.655696 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.655654 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:53:41.874197 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.869996 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:41.998222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.974846 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" event={"ID":"ff075586-a6a8-4eed-819d-19c55abfa057","Type":"ContainerStarted","Data":"0168693482f2b152302e51f5be14af7f8b236c6d8cf95c3a4e77d193457074bd"} Oct 30 01:53:41.998222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:41.974905 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" event={"ID":"ff075586-a6a8-4eed-819d-19c55abfa057","Type":"ContainerStarted","Data":"ad498602b214adfd921eb56b24677ffb51eb15416e91ae584476c40429433b11"} Oct 30 01:53:42.257289 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:42.241167 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:42.319116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:42.318113 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:42.319116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:42.318261 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:53:42.319116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:42.318304 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" podUID="ff075586-a6a8-4eed-819d-19c55abfa057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 01:53:42.454114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:42.439849 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:53:42.454114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:42.439928 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" podUID="d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 01:53:43.091262 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:43.091202 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:43.311778 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:43.311731 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:53:43.312063 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:43.312021 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" podUID="ff075586-a6a8-4eed-819d-19c55abfa057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 01:53:43.436522 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:43.436476 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:53:43.436732 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:43.436544 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" podUID="d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 01:53:43.515606 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:43.515559 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:43.797301 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:43.797194 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:43.808910 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:43.808223 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:43.831302 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:43.831242 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:43.850296 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:43.850238 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:43.850957 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:43.850925 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:43.871256 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:43.871172 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 01:53:43.904274 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:43.904230 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" podUID="f02332d2-b193-4f97-bb47-d0d0904bdc9a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:44.026459 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:44.026415 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:44.096172 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:44.095103 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:44.104585 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:44.104552 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:53:44.181934 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:44.181874 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" podStartSLOduration=4.181852847 podStartE2EDuration="4.181852847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:42.084890411 +0000 UTC m=+1451.663681516" watchObservedRunningTime="2026-10-30 01:53:44.181852847 +0000 UTC m=+1453.760643929" Oct 30 01:53:44.333116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:44.317072 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:44.363351 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:44.353855 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:53:44.363673 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:44.363637 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" podUID="ff075586-a6a8-4eed-819d-19c55abfa057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 01:53:44.448452 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:44.436229 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:53:44.448452 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:44.436303 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" podUID="d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 01:53:44.459698 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:44.459404 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" podUID="f02332d2-b193-4f97-bb47-d0d0904bdc9a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.27:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:44.806116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:44.803219 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 01:53:44.806116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:44.803686 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:53:44.866240 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:44.862819 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 01:53:44.866240 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:44.863249 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 01:53:44.882114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:44.878254 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:53:44.911175 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:44.911133 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:53:45.132187 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.103239 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 01:53:45.333273 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.333232 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:53:45.333538 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.333508 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" podUID="6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:45.334290 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.334260 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:53:45.334483 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.334449 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" podUID="ff075586-a6a8-4eed-819d-19c55abfa057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 01:53:45.372192 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.372147 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw"] Oct 30 01:53:45.445637 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.433055 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw" Oct 30 01:53:45.602116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.512503 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:53:45.602116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.601263 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" podUID="d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 01:53:45.602784 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.512610 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:53:45.761295 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.761201 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" podUID="e34d09e7-ec9d-4160-9c74-13b2ccf8d871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:45.761542 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.622706 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:53:45.761654 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.622413 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:53:45.761758 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.623005 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:53:45.765477 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.623274 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:53:45.790127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.766879 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw"] Oct 30 01:53:45.790127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.731673 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjzd6\" (UniqueName: \"kubernetes.io/projected/7239897c-d0c8-4367-8ec3-cb0320eab379-kube-api-access-pjzd6\") pod \"maas-api-revoke-sub-a2de2e31-tgcqw\" (UID: \"7239897c-d0c8-4367-8ec3-cb0320eab379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw" Oct 30 01:53:45.790127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.780120 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7239897c-d0c8-4367-8ec3-cb0320eab379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2de2e31-tgcqw\" (UID: \"7239897c-d0c8-4367-8ec3-cb0320eab379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw" Oct 30 01:53:45.790127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.780292 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7239897c-d0c8-4367-8ec3-cb0320eab379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2de2e31-tgcqw\" (UID: \"7239897c-d0c8-4367-8ec3-cb0320eab379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw" Oct 30 01:53:45.810471 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.810362 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:45.854174 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.824282 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 01:53:45.871583 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.855328 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:45.872466 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.870194 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:45.872946 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.870228 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 01:53:45.883228 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.883181 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:45.925120 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.913061 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7239897c-d0c8-4367-8ec3-cb0320eab379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2de2e31-tgcqw\" (UID: \"7239897c-d0c8-4367-8ec3-cb0320eab379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw" Oct 30 01:53:45.925120 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.913155 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7239897c-d0c8-4367-8ec3-cb0320eab379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2de2e31-tgcqw\" (UID: \"7239897c-d0c8-4367-8ec3-cb0320eab379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw" Oct 30 01:53:45.925120 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.913376 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjzd6\" (UniqueName: \"kubernetes.io/projected/7239897c-d0c8-4367-8ec3-cb0320eab379-kube-api-access-pjzd6\") pod \"maas-api-revoke-sub-a2de2e31-tgcqw\" (UID: \"7239897c-d0c8-4367-8ec3-cb0320eab379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw" Oct 30 01:53:45.925120 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:45.914874 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7239897c-d0c8-4367-8ec3-cb0320eab379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2de2e31-tgcqw\" (UID: \"7239897c-d0c8-4367-8ec3-cb0320eab379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw" Oct 30 01:53:46.044322 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:46.044212 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:46.106312 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:46.106268 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:46.183112 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:46.170832 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7239897c-d0c8-4367-8ec3-cb0320eab379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2de2e31-tgcqw\" (UID: \"7239897c-d0c8-4367-8ec3-cb0320eab379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw" Oct 30 01:53:46.187218 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:46.187186 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjzd6\" (UniqueName: \"kubernetes.io/projected/7239897c-d0c8-4367-8ec3-cb0320eab379-kube-api-access-pjzd6\") pod \"maas-api-revoke-sub-a2de2e31-tgcqw\" (UID: \"7239897c-d0c8-4367-8ec3-cb0320eab379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw" Oct 30 01:53:46.310504 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:46.310404 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:53:46.343127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:46.314192 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" podUID="ff075586-a6a8-4eed-819d-19c55abfa057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 01:53:46.487119 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:46.486246 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:53:46.500121 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:46.486345 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" podUID="d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 01:53:46.510116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:46.502778 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw" Oct 30 01:53:46.832512 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:46.832462 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:46.941176 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:46.884302 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:46.941176 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:46.884707 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 01:53:46.941176 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:46.886237 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:46.941176 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:46.886589 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:53:47.094113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:46.996494 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:47.094113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:47.015751 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:47.094113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:47.067324 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:47.094113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:47.068121 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:53:47.179835 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:47.162809 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:47.276035 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:47.275930 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:53:47.377062 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:47.377019 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:53:47.395403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:47.377478 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" podUID="ff075586-a6a8-4eed-819d-19c55abfa057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 01:53:47.395403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:47.385456 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:53:47.479667 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:47.479633 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:53:47.479962 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:47.479916 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" podUID="d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 01:53:47.645356 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:47.645266 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:53:47.680655 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:47.680609 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:47.779131 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:47.769675 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:47.795360 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:47.795330 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:53:47.842696 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:47.816187 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:47.844225 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:47.842995 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:47.907271 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:47.907175 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:47.999923 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:47.999894 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:53:48.036020 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:48.035981 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:48.098937 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:48.098905 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:53:48.288135 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:48.286603 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw"] Oct 30 01:53:48.291962 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:53:48.291916 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7239897c_d0c8_4367_8ec3_cb0320eab379.slice/crio-415868fcc8cc2903eb3caefa728557dca6d0c6b397398dd252bb574c34ea3d5f WatchSource:0}: Error finding container 415868fcc8cc2903eb3caefa728557dca6d0c6b397398dd252bb574c34ea3d5f: Status 404 returned error can't find the container with id 415868fcc8cc2903eb3caefa728557dca6d0c6b397398dd252bb574c34ea3d5f Oct 30 01:53:48.370175 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:48.358201 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:53:48.370175 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:48.358263 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" podUID="ff075586-a6a8-4eed-819d-19c55abfa057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 01:53:48.451138 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:48.449496 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:53:48.451138 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:48.449561 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" podUID="d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 01:53:48.960163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:48.957147 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:48.975763 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:48.975732 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw" event={"ID":"7239897c-d0c8-4367-8ec3-cb0320eab379","Type":"ContainerStarted","Data":"415868fcc8cc2903eb3caefa728557dca6d0c6b397398dd252bb574c34ea3d5f"} Oct 30 01:53:49.084249 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:49.084122 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:49.254126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:49.249653 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:49.311115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:49.310575 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:53:49.311115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:49.310652 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" podUID="ff075586-a6a8-4eed-819d-19c55abfa057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 01:53:49.441116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:49.435627 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:53:49.441116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:49.435707 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" podUID="d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 01:53:49.863960 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:49.863925 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:49.885473 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:49.885438 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:49.988751 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:49.988659 5998 generic.go:356] "Generic (PLEG): container finished" podID="7239897c-d0c8-4367-8ec3-cb0320eab379" containerID="046b87f3bf629f27b063686cfbd200299fe32c810dad7c40b05839fcb75eb540" exitCode=0 Oct 30 01:53:49.988975 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:49.988951 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw" event={"ID":"7239897c-d0c8-4367-8ec3-cb0320eab379","Type":"ContainerDied","Data":"046b87f3bf629f27b063686cfbd200299fe32c810dad7c40b05839fcb75eb540"} Oct 30 01:53:50.056013 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.055969 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:53:50.324169 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.317523 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:53:50.324169 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.317586 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" podUID="ff075586-a6a8-4eed-819d-19c55abfa057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 01:53:50.434546 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.434511 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:53:50.434751 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.434564 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" podUID="d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 01:53:50.491167 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.491135 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz"] Oct 30 01:53:50.495322 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.495296 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz" Oct 30 01:53:50.551601 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.551573 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz"] Oct 30 01:53:50.579466 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.579380 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f869d79-tgbxz\" (UID: \"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz" Oct 30 01:53:50.579466 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.579433 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hrjd\" (UniqueName: \"kubernetes.io/projected/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-kube-api-access-2hrjd\") pod \"maas-api-revoke-sub-3f869d79-tgbxz\" (UID: \"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz" Oct 30 01:53:50.579631 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.579526 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f869d79-tgbxz\" (UID: \"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz" Oct 30 01:53:50.627179 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.627140 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:50.681238 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.681180 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f869d79-tgbxz\" (UID: \"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz" Oct 30 01:53:50.681452 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.681263 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hrjd\" (UniqueName: \"kubernetes.io/projected/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-kube-api-access-2hrjd\") pod \"maas-api-revoke-sub-3f869d79-tgbxz\" (UID: \"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz" Oct 30 01:53:50.681452 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.681310 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f869d79-tgbxz\" (UID: \"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz" Oct 30 01:53:50.682259 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.682229 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f869d79-tgbxz\" (UID: \"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz" Oct 30 01:53:50.684127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.684100 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f869d79-tgbxz\" (UID: \"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz" Oct 30 01:53:50.748117 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.738431 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hrjd\" (UniqueName: \"kubernetes.io/projected/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-kube-api-access-2hrjd\") pod \"maas-api-revoke-sub-3f869d79-tgbxz\" (UID: \"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz" Oct 30 01:53:50.851286 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:50.851202 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz" Oct 30 01:53:51.222110 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:51.222029 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz"] Oct 30 01:53:51.226035 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:53:51.225223 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69fea0c4_aa7f_47e8_b1d8_ceaf031a20a2.slice/crio-f4cdb4c69fe14d9a1eb96c8fe164bf6d655470d4f949b3f67bd02d4ddcbbc9ce WatchSource:0}: Error finding container f4cdb4c69fe14d9a1eb96c8fe164bf6d655470d4f949b3f67bd02d4ddcbbc9ce: Status 404 returned error can't find the container with id f4cdb4c69fe14d9a1eb96c8fe164bf6d655470d4f949b3f67bd02d4ddcbbc9ce Oct 30 01:53:51.260483 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:51.260452 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw" Oct 30 01:53:51.313149 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:51.312633 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:53:51.313149 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:51.312696 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" podUID="ff075586-a6a8-4eed-819d-19c55abfa057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 01:53:51.389615 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:51.389573 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7239897c-d0c8-4367-8ec3-cb0320eab379-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7239897c-d0c8-4367-8ec3-cb0320eab379-maas-api-xks-ca\") pod \"7239897c-d0c8-4367-8ec3-cb0320eab379\" (UID: \"7239897c-d0c8-4367-8ec3-cb0320eab379\") " Oct 30 01:53:51.390176 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:51.389719 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7239897c-d0c8-4367-8ec3-cb0320eab379-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7239897c-d0c8-4367-8ec3-cb0320eab379-maas-api-openshift-ca\") pod \"7239897c-d0c8-4367-8ec3-cb0320eab379\" (UID: \"7239897c-d0c8-4367-8ec3-cb0320eab379\") " Oct 30 01:53:51.390176 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:51.389820 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7239897c-d0c8-4367-8ec3-cb0320eab379-kube-api-access-pjzd6\" (UniqueName: \"kubernetes.io/projected/7239897c-d0c8-4367-8ec3-cb0320eab379-kube-api-access-pjzd6\") pod \"7239897c-d0c8-4367-8ec3-cb0320eab379\" (UID: \"7239897c-d0c8-4367-8ec3-cb0320eab379\") " Oct 30 01:53:51.390331 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:51.390301 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7239897c-d0c8-4367-8ec3-cb0320eab379-maas-api-openshift-ca" pod "7239897c-d0c8-4367-8ec3-cb0320eab379" (UID: "7239897c-d0c8-4367-8ec3-cb0320eab379"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:51.393909 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:51.393872 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7239897c-d0c8-4367-8ec3-cb0320eab379-kube-api-access-pjzd6" pod "7239897c-d0c8-4367-8ec3-cb0320eab379" (UID: "7239897c-d0c8-4367-8ec3-cb0320eab379"). InnerVolumeSpecName "kube-api-access-pjzd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:51.394067 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:51.393963 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7239897c-d0c8-4367-8ec3-cb0320eab379-maas-api-xks-ca" pod "7239897c-d0c8-4367-8ec3-cb0320eab379" (UID: "7239897c-d0c8-4367-8ec3-cb0320eab379"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:51.492118 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:51.492055 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7239897c-d0c8-4367-8ec3-cb0320eab379-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:51.492305 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:51.492109 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjzd6\" (UniqueName: \"kubernetes.io/projected/7239897c-d0c8-4367-8ec3-cb0320eab379-kube-api-access-pjzd6\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:51.492305 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:51.492150 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7239897c-d0c8-4367-8ec3-cb0320eab379-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:51.782958 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:51.782926 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:51.783460 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:51.783436 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:51.784376 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:51.784356 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 01:53:52.033654 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:52.028598 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz" event={"ID":"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2","Type":"ContainerStarted","Data":"253118fb5e4b060d257828eedcbe9dcb8fafc81bce932fde79414c069de98dc5"} Oct 30 01:53:52.033654 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:52.028666 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz" event={"ID":"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2","Type":"ContainerStarted","Data":"f4cdb4c69fe14d9a1eb96c8fe164bf6d655470d4f949b3f67bd02d4ddcbbc9ce"} Oct 30 01:53:52.034287 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:52.033921 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw" Oct 30 01:53:52.034287 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:52.034211 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw" event={"ID":"7239897c-d0c8-4367-8ec3-cb0320eab379","Type":"ContainerDied","Data":"415868fcc8cc2903eb3caefa728557dca6d0c6b397398dd252bb574c34ea3d5f"} Oct 30 01:53:52.034287 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:52.034267 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="415868fcc8cc2903eb3caefa728557dca6d0c6b397398dd252bb574c34ea3d5f" Oct 30 01:53:52.071147 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:52.071066 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz" podStartSLOduration=2.07104611 podStartE2EDuration="2.07104611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:52.070053157 +0000 UTC m=+1461.648844239" watchObservedRunningTime="2026-10-30 01:53:52.07104611 +0000 UTC m=+1461.649837192" Oct 30 01:53:52.268583 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:52.268541 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:52.419034 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:52.418986 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:52.420934 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:52.420901 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:52.424553 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:52.423666 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 01:53:53.046388 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:53.046346 5998 generic.go:356] "Generic (PLEG): container finished" podID="69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2" containerID="253118fb5e4b060d257828eedcbe9dcb8fafc81bce932fde79414c069de98dc5" exitCode=0 Oct 30 01:53:53.046664 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:53.046435 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz" event={"ID":"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2","Type":"ContainerDied","Data":"253118fb5e4b060d257828eedcbe9dcb8fafc81bce932fde79414c069de98dc5"} Oct 30 01:53:53.980342 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:53.980297 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:54.226193 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:54.226159 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz" Oct 30 01:53:54.341407 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:54.341328 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-maas-api-openshift-ca\") pod \"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2\" (UID: \"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2\") " Oct 30 01:53:54.341556 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:54.341508 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-kube-api-access-2hrjd\" (UniqueName: \"kubernetes.io/projected/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-kube-api-access-2hrjd\") pod \"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2\" (UID: \"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2\") " Oct 30 01:53:54.341674 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:54.341655 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-maas-api-xks-ca\") pod \"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2\" (UID: \"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2\") " Oct 30 01:53:54.341752 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:54.341727 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-maas-api-openshift-ca" pod "69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2" (UID: "69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:54.342054 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:54.342034 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:54.343657 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:54.343632 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-kube-api-access-2hrjd" pod "69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2" (UID: "69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2"). InnerVolumeSpecName "kube-api-access-2hrjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:54.343657 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:54.343635 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-maas-api-xks-ca" pod "69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2" (UID: "69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:54.443310 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:54.443268 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hrjd\" (UniqueName: \"kubernetes.io/projected/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-kube-api-access-2hrjd\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:54.443310 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:54.443303 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:53:55.065208 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:55.065176 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz" Oct 30 01:53:55.065640 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:55.065203 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz" event={"ID":"69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2","Type":"ContainerDied","Data":"f4cdb4c69fe14d9a1eb96c8fe164bf6d655470d4f949b3f67bd02d4ddcbbc9ce"} Oct 30 01:53:55.065640 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:55.065257 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4cdb4c69fe14d9a1eb96c8fe164bf6d655470d4f949b3f67bd02d4ddcbbc9ce" Oct 30 01:53:55.217810 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:53:55.217766 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:53:55.609422 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:55.609378 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:56.295113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:56.295044 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:57.305815 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:57.305776 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:57.607542 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:57.607443 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:57.757120 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:57.757068 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:58.290860 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.290826 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:58.514414 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.514378 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:58.646805 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.646706 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn"] Oct 30 01:53:58.648073 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.648032 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7239897c-d0c8-4367-8ec3-cb0320eab379" containerName="revoke-keys" Oct 30 01:53:58.648073 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.648062 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7239897c-d0c8-4367-8ec3-cb0320eab379" containerName="revoke-keys" Oct 30 01:53:58.656337 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.656306 5998 cpu_manager.go:414] "containerMap: removing container" podUID="69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2" containerName="revoke-keys" Oct 30 01:53:58.656561 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.656544 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2" containerName="revoke-keys" Oct 30 01:53:58.657083 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.657048 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2" containerName="revoke-keys" Oct 30 01:53:58.657083 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.657108 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7239897c-d0c8-4367-8ec3-cb0320eab379" containerName="revoke-keys" Oct 30 01:53:58.678118 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.676047 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn"] Oct 30 01:53:58.678118 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.676253 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn" Oct 30 01:53:58.696831 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.696432 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:53:58.708262 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.708230 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:53:58.708426 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.708294 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:53:58.708426 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.708341 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:53:58.712247 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.712218 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnzp9\" (UniqueName: \"kubernetes.io/projected/f51353d2-1f84-494e-829f-1d7eb6ed14d1-kube-api-access-pnzp9\") pod \"maas-api-revoke-sub-22c47e8f-pv9wn\" (UID: \"f51353d2-1f84-494e-829f-1d7eb6ed14d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn" Oct 30 01:53:58.712377 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.712341 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51353d2-1f84-494e-829f-1d7eb6ed14d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c47e8f-pv9wn\" (UID: \"f51353d2-1f84-494e-829f-1d7eb6ed14d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn" Oct 30 01:53:58.712451 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.712411 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51353d2-1f84-494e-829f-1d7eb6ed14d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c47e8f-pv9wn\" (UID: \"f51353d2-1f84-494e-829f-1d7eb6ed14d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn" Oct 30 01:53:58.819932 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.817115 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51353d2-1f84-494e-829f-1d7eb6ed14d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c47e8f-pv9wn\" (UID: \"f51353d2-1f84-494e-829f-1d7eb6ed14d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn" Oct 30 01:53:58.819932 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.817207 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51353d2-1f84-494e-829f-1d7eb6ed14d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c47e8f-pv9wn\" (UID: \"f51353d2-1f84-494e-829f-1d7eb6ed14d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn" Oct 30 01:53:58.819932 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.817338 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnzp9\" (UniqueName: \"kubernetes.io/projected/f51353d2-1f84-494e-829f-1d7eb6ed14d1-kube-api-access-pnzp9\") pod \"maas-api-revoke-sub-22c47e8f-pv9wn\" (UID: \"f51353d2-1f84-494e-829f-1d7eb6ed14d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn" Oct 30 01:53:58.819932 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.818737 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51353d2-1f84-494e-829f-1d7eb6ed14d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22c47e8f-pv9wn\" (UID: \"f51353d2-1f84-494e-829f-1d7eb6ed14d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn" Oct 30 01:53:58.829947 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.829911 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51353d2-1f84-494e-829f-1d7eb6ed14d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22c47e8f-pv9wn\" (UID: \"f51353d2-1f84-494e-829f-1d7eb6ed14d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn" Oct 30 01:53:58.850932 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.850888 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnzp9\" (UniqueName: \"kubernetes.io/projected/f51353d2-1f84-494e-829f-1d7eb6ed14d1-kube-api-access-pnzp9\") pod \"maas-api-revoke-sub-22c47e8f-pv9wn\" (UID: \"f51353d2-1f84-494e-829f-1d7eb6ed14d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn" Oct 30 01:53:58.944311 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:58.944273 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:59.012020 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:59.011984 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn" Oct 30 01:53:59.179512 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:59.179480 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:59.324150 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:59.323078 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn"] Oct 30 01:53:59.325063 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:53:59.325026 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51353d2_1f84_494e_829f_1d7eb6ed14d1.slice/crio-7571961be36fe1a9592229ea1578026cf90d7162eb3aea10f7f1eacd51cc44e0 WatchSource:0}: Error finding container 7571961be36fe1a9592229ea1578026cf90d7162eb3aea10f7f1eacd51cc44e0: Status 404 returned error can't find the container with id 7571961be36fe1a9592229ea1578026cf90d7162eb3aea10f7f1eacd51cc44e0 Oct 30 01:53:59.457530 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:59.457488 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:53:59.650327 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:53:59.650296 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:00.110951 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:00.110903 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn" event={"ID":"f51353d2-1f84-494e-829f-1d7eb6ed14d1","Type":"ContainerStarted","Data":"c4c947996d97740bc2b8836c7bfcfa81b671002c8c24186cec20750d9748ded1"} Oct 30 01:54:00.111182 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:00.110971 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn" event={"ID":"f51353d2-1f84-494e-829f-1d7eb6ed14d1","Type":"ContainerStarted","Data":"7571961be36fe1a9592229ea1578026cf90d7162eb3aea10f7f1eacd51cc44e0"} Oct 30 01:54:00.155473 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:00.155397 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn" podStartSLOduration=2.1553807 podStartE2EDuration="2.1553807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:00.15261303 +0000 UTC m=+1469.731404112" watchObservedRunningTime="2026-10-30 01:54:00.1553807 +0000 UTC m=+1469.734171780" Oct 30 01:54:00.923079 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:00.923028 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:01.168723 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:01.168687 5998 generic.go:356] "Generic (PLEG): container finished" podID="f51353d2-1f84-494e-829f-1d7eb6ed14d1" containerID="c4c947996d97740bc2b8836c7bfcfa81b671002c8c24186cec20750d9748ded1" exitCode=0 Oct 30 01:54:01.169127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:01.169081 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn" event={"ID":"f51353d2-1f84-494e-829f-1d7eb6ed14d1","Type":"ContainerDied","Data":"c4c947996d97740bc2b8836c7bfcfa81b671002c8c24186cec20750d9748ded1"} Oct 30 01:54:01.535125 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:01.534409 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:02.650174 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:02.628728 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:02.697246 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:02.697199 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:02.780121 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:02.780069 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:02.803113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:02.802592 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 01:54:02.827115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:02.817190 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:02.863475 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:02.863162 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:02.970116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:02.969188 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:03.027121 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.017197 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:03.176116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.162238 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:03.508336 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.507941 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn" Oct 30 01:54:03.542117 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.509269 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:03.667180 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.640705 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f51353d2-1f84-494e-829f-1d7eb6ed14d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51353d2-1f84-494e-829f-1d7eb6ed14d1-maas-api-xks-ca\") pod \"f51353d2-1f84-494e-829f-1d7eb6ed14d1\" (UID: \"f51353d2-1f84-494e-829f-1d7eb6ed14d1\") " Oct 30 01:54:03.678244 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.678205 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51353d2-1f84-494e-829f-1d7eb6ed14d1-kube-api-access-pnzp9\" (UniqueName: \"kubernetes.io/projected/f51353d2-1f84-494e-829f-1d7eb6ed14d1-kube-api-access-pnzp9\") pod \"f51353d2-1f84-494e-829f-1d7eb6ed14d1\" (UID: \"f51353d2-1f84-494e-829f-1d7eb6ed14d1\") " Oct 30 01:54:03.684264 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.684229 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f51353d2-1f84-494e-829f-1d7eb6ed14d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51353d2-1f84-494e-829f-1d7eb6ed14d1-maas-api-openshift-ca\") pod \"f51353d2-1f84-494e-829f-1d7eb6ed14d1\" (UID: \"f51353d2-1f84-494e-829f-1d7eb6ed14d1\") " Oct 30 01:54:03.753115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.671216 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:54:03.757117 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.753403 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" podUID="592f8d79-2a33-4c5b-9639-c60bd7aedf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:03.761220 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.738843 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:03.765260 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.698605 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 01:54:03.769203 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.697518 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51353d2-1f84-494e-829f-1d7eb6ed14d1-maas-api-openshift-ca" pod "f51353d2-1f84-494e-829f-1d7eb6ed14d1" (UID: "f51353d2-1f84-494e-829f-1d7eb6ed14d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:54:03.770349 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.770263 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51353d2-1f84-494e-829f-1d7eb6ed14d1-kube-api-access-pnzp9" pod "f51353d2-1f84-494e-829f-1d7eb6ed14d1" (UID: "f51353d2-1f84-494e-829f-1d7eb6ed14d1"). InnerVolumeSpecName "kube-api-access-pnzp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:54:03.771687 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.770564 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:54:03.784114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.782225 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f51353d2-1f84-494e-829f-1d7eb6ed14d1-maas-api-xks-ca" pod "f51353d2-1f84-494e-829f-1d7eb6ed14d1" (UID: "f51353d2-1f84-494e-829f-1d7eb6ed14d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:54:03.822793 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.822747 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:03.823223 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.823185 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:54:03.824350 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.824049 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:03.824540 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.824515 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 01:54:03.825671 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.825647 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:54:03.851364 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.849609 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f51353d2-1f84-494e-829f-1d7eb6ed14d1-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:54:03.851364 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.849679 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnzp9\" (UniqueName: \"kubernetes.io/projected/f51353d2-1f84-494e-829f-1d7eb6ed14d1-kube-api-access-pnzp9\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:54:03.851364 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.849696 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f51353d2-1f84-494e-829f-1d7eb6ed14d1-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:54:03.891736 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.870739 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 01:54:03.891736 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.874279 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:54:03.929764 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.909355 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:03.929764 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:03.926227 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" podUID="f02332d2-b193-4f97-bb47-d0d0904bdc9a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:04.047148 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:04.039469 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 01:54:04.055381 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:04.055344 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:54:04.109358 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:04.062389 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:04.110109 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:04.110014 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:04.179960 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:04.179855 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 01:54:04.180580 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:04.180554 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:54:04.240297 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:04.240247 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:54:04.248129 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:04.240653 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" podUID="43cfa231-6176-4512-9307-aa872cbd3f6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:04.367127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:04.359555 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:04.551180 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:04.544188 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" podUID="f02332d2-b193-4f97-bb47-d0d0904bdc9a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.27:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:04.759145 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:04.720892 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn" Oct 30 01:54:04.846557 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:04.846433 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:04.848440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:04.848395 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 01:54:04.853113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:04.849195 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:04.853113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:04.849486 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:04.934117 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:04.913382 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:04.934117 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:04.914308 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 01:54:05.017811 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:05.017671 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:54:05.017811 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:05.017737 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" podUID="41479a83-7edc-4856-b62c-e12ee3c2d814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:05.048148 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:05.047781 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn" event={"ID":"f51353d2-1f84-494e-829f-1d7eb6ed14d1","Type":"ContainerDied","Data":"7571961be36fe1a9592229ea1578026cf90d7162eb3aea10f7f1eacd51cc44e0"} Oct 30 01:54:05.048148 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:05.047849 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7571961be36fe1a9592229ea1578026cf90d7162eb3aea10f7f1eacd51cc44e0" Oct 30 01:54:05.048148 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:05.047914 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:54:05.048148 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:05.047964 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:54:05.056129 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:05.055531 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 01:54:05.082133 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:05.081183 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 01:54:05.082133 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:05.081617 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:54:05.242378 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:05.242287 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:05.540172 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:05.536387 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:54:05.909379 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:05.909224 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 01:54:05.911512 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:05.911383 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 01:54:05.911946 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:05.911920 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 01:54:05.960284 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:05.939026 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:54:05.993910 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:05.993881 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:54:06.023314 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:06.013434 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:54:06.051491 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:06.051461 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:54:06.072114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:06.069212 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:54:06.089140 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:06.087560 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 01:54:06.553280 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:06.553243 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:06.784049 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:06.783946 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:07.107116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:07.015664 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 01:54:07.108874 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:07.031647 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:07.109458 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:07.088212 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:07.127847 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:07.127809 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:07.592116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:07.573818 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:54:07.662896 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:07.662858 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:07.915366 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:07.915336 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:54:07.998072 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:07.998042 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:54:09.663450 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:09.663414 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:09.773955 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:09.773896 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:10.112325 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:10.112229 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:10.242887 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:10.242855 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:11.979558 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:11.979526 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:13.676414 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:13.676378 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:15.208053 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:15.208012 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:16.772389 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:16.772353 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:16.882785 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:16.882735 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:18.007398 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:18.007360 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:54:18.631462 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:18.631421 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:20.543201 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:20.542332 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:20.582124 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:20.581185 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 01:54:20.863297 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:20.863191 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:20.898429 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:20.898372 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:20.918876 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:20.918832 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:20.920317 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:20.919324 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:21.054272 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:21.054218 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:21.060206 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:21.060165 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:21.092237 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:21.092187 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 01:54:21.245252 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:21.245195 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:21.546563 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:21.546464 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:21.600761 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:21.600729 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:54:21.626244 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:21.606183 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:21.626244 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:21.621049 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:21.626244 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:21.625984 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:54:21.954703 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:21.954661 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 01:54:21.955669 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:21.955635 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:54:21.975908 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:21.975822 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:21.976226 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:21.976206 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:54:21.985710 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:21.985676 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:54:22.101136 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:22.100639 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:54:22.126738 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:22.126590 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:54:22.152706 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:22.151369 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:54:22.556988 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:22.556946 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:24.440750 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.440714 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:24.586320 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.586288 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62"] Oct 30 01:54:24.586830 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.586813 5998 cpu_manager.go:414] "containerMap: removing container" podUID="f51353d2-1f84-494e-829f-1d7eb6ed14d1" containerName="revoke-keys" Oct 30 01:54:24.586830 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.586828 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51353d2-1f84-494e-829f-1d7eb6ed14d1" containerName="revoke-keys" Oct 30 01:54:24.586976 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.586941 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51353d2-1f84-494e-829f-1d7eb6ed14d1" containerName="revoke-keys" Oct 30 01:54:24.591063 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.591039 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62" Oct 30 01:54:24.613701 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.613671 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62"] Oct 30 01:54:24.623727 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.619291 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:54:24.623727 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.619487 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:54:24.623727 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.620573 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:54:24.624082 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.624056 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:54:24.723494 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.723382 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nld8\" (UniqueName: \"kubernetes.io/projected/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-kube-api-access-2nld8\") pod \"maas-api-revoke-sub-48b7ef7b-l5x62\" (UID: \"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62" Oct 30 01:54:24.723681 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.723510 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48b7ef7b-l5x62\" (UID: \"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62" Oct 30 01:54:24.723681 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.723664 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48b7ef7b-l5x62\" (UID: \"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62" Oct 30 01:54:24.824613 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.824565 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nld8\" (UniqueName: \"kubernetes.io/projected/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-kube-api-access-2nld8\") pod \"maas-api-revoke-sub-48b7ef7b-l5x62\" (UID: \"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62" Oct 30 01:54:24.824814 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.824629 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48b7ef7b-l5x62\" (UID: \"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62" Oct 30 01:54:24.824814 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.824803 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48b7ef7b-l5x62\" (UID: \"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62" Oct 30 01:54:24.825492 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.825467 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48b7ef7b-l5x62\" (UID: \"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62" Oct 30 01:54:24.827129 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.827083 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48b7ef7b-l5x62\" (UID: \"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62" Oct 30 01:54:24.847619 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.847585 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nld8\" (UniqueName: \"kubernetes.io/projected/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-kube-api-access-2nld8\") pod \"maas-api-revoke-sub-48b7ef7b-l5x62\" (UID: \"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62" Oct 30 01:54:24.904659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:24.904614 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62" Oct 30 01:54:25.218674 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:25.217860 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62"] Oct 30 01:54:25.220690 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:54:25.220661 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf721b93a_2a8e_4aaf_8e6a_6df0318d4fd1.slice/crio-47a474f3d4e2c55925f5c34b0a9ecd7ac4fe902d61a7bfea5a22fc3c6db88603 WatchSource:0}: Error finding container 47a474f3d4e2c55925f5c34b0a9ecd7ac4fe902d61a7bfea5a22fc3c6db88603: Status 404 returned error can't find the container with id 47a474f3d4e2c55925f5c34b0a9ecd7ac4fe902d61a7bfea5a22fc3c6db88603 Oct 30 01:54:25.233814 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:54:25.233763 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:54:25.526783 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:25.526748 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62" event={"ID":"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1","Type":"ContainerStarted","Data":"b45b6d19b00ab4d3ae3964b4d61f8b237e7d91b43caf4079048f1641006d8da1"} Oct 30 01:54:25.527285 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:25.526810 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62" event={"ID":"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1","Type":"ContainerStarted","Data":"47a474f3d4e2c55925f5c34b0a9ecd7ac4fe902d61a7bfea5a22fc3c6db88603"} Oct 30 01:54:25.572101 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:25.572046 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62" podStartSLOduration=1.572030547 podStartE2EDuration="1.572030547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:25.568659437 +0000 UTC m=+1495.147450518" watchObservedRunningTime="2026-10-30 01:54:25.572030547 +0000 UTC m=+1495.150821629" Oct 30 01:54:26.113839 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.113800 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:26.336489 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.336411 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5"] Oct 30 01:54:26.340590 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.340568 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.358457 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.358424 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq"] Oct 30 01:54:26.362269 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.362251 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.370296 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.370267 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5"] Oct 30 01:54:26.396146 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.396084 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq"] Oct 30 01:54:26.448723 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.448690 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.448903 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.448742 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd04c1d-6bb4-4cd8-99b5-523538908019-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.448903 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.448883 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.448980 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.448960 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.449014 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.448993 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.449056 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.449018 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.449128 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.449114 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.449176 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.449148 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.449176 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.449168 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.449240 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.449226 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdx77\" (UniqueName: \"kubernetes.io/projected/6cd04c1d-6bb4-4cd8-99b5-523538908019-kube-api-access-jdx77\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.449320 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.449273 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7wtf\" (UniqueName: \"kubernetes.io/projected/63830843-4d31-45a7-83c9-9bb2248abe3b-kube-api-access-l7wtf\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.449403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.449330 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.449403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.449353 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.449403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.449382 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.449515 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.449441 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63830843-4d31-45a7-83c9-9bb2248abe3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.449515 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.449492 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.449581 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.449540 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.449581 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.449566 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.535398 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.535362 5998 generic.go:356] "Generic (PLEG): container finished" podID="f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1" containerID="b45b6d19b00ab4d3ae3964b4d61f8b237e7d91b43caf4079048f1641006d8da1" exitCode=0 Oct 30 01:54:26.539053 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.539019 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62" event={"ID":"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1","Type":"ContainerDied","Data":"b45b6d19b00ab4d3ae3964b4d61f8b237e7d91b43caf4079048f1641006d8da1"} Oct 30 01:54:26.550682 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.550652 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.550849 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.550693 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.550849 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.550721 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.550849 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.550758 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.550849 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.550790 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.550849 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.550815 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.551119 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.550853 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdx77\" (UniqueName: \"kubernetes.io/projected/6cd04c1d-6bb4-4cd8-99b5-523538908019-kube-api-access-jdx77\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.551119 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.550895 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7wtf\" (UniqueName: \"kubernetes.io/projected/63830843-4d31-45a7-83c9-9bb2248abe3b-kube-api-access-l7wtf\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.551119 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.550936 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.551119 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.550970 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.551119 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.551015 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.551119 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.551049 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63830843-4d31-45a7-83c9-9bb2248abe3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.551119 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.551102 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.551491 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.551108 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.551491 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.551185 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.551491 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.551215 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.551491 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.551248 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.551491 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.551288 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.551491 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.551331 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd04c1d-6bb4-4cd8-99b5-523538908019-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.551491 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.551360 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.551491 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.551411 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.551913 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.551654 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.551913 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.551664 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.551913 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.551737 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.551913 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.551744 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.551913 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.551894 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.552781 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.552752 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63830843-4d31-45a7-83c9-9bb2248abe3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.552884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.552758 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd04c1d-6bb4-4cd8-99b5-523538908019-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.553558 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.553521 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.553698 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.553677 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.554045 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.554026 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.554250 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.554234 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.576669 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.576640 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.577178 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.577146 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdx77\" (UniqueName: \"kubernetes.io/projected/6cd04c1d-6bb4-4cd8-99b5-523538908019-kube-api-access-jdx77\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.578435 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.578408 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7wtf\" (UniqueName: \"kubernetes.io/projected/63830843-4d31-45a7-83c9-9bb2248abe3b-kube-api-access-l7wtf\") pod \"maas-default-gateway-openshift-default-75d7c785f-kf6tq\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.578807 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.578783 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-sdrx5\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.656537 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.656457 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:26.678168 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.678134 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:26.983249 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.983221 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5"] Oct 30 01:54:26.985607 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:54:26.985578 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd04c1d_6bb4_4cd8_99b5_523538908019.slice/crio-fe3ceb4da20823f6885f60cdfbcdcfdb52caf7c3d41f2f15efe745ab7ccb0c3a WatchSource:0}: Error finding container fe3ceb4da20823f6885f60cdfbcdcfdb52caf7c3d41f2f15efe745ab7ccb0c3a: Status 404 returned error can't find the container with id fe3ceb4da20823f6885f60cdfbcdcfdb52caf7c3d41f2f15efe745ab7ccb0c3a Oct 30 01:54:26.988565 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.988529 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:54:26.988656 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.988613 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:54:26.988723 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:26.988659 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:54:27.006107 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.006066 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq"] Oct 30 01:54:27.014700 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:54:27.013447 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63830843_4d31_45a7_83c9_9bb2248abe3b.slice/crio-c3589b335c2f0748c19ef15df2c99dd3e03c09806aa654457efdbdf9cbdef5d6 WatchSource:0}: Error finding container c3589b335c2f0748c19ef15df2c99dd3e03c09806aa654457efdbdf9cbdef5d6: Status 404 returned error can't find the container with id c3589b335c2f0748c19ef15df2c99dd3e03c09806aa654457efdbdf9cbdef5d6 Oct 30 01:54:27.015882 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.015843 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:54:27.015989 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.015936 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:54:27.015989 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.015976 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:54:27.546312 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.546272 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" event={"ID":"6cd04c1d-6bb4-4cd8-99b5-523538908019","Type":"ContainerStarted","Data":"a910a7c81dc5da24505757cd5dabbad04e5a5cdd19a3122057bdef721cae0c85"} Oct 30 01:54:27.546806 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.546339 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" event={"ID":"6cd04c1d-6bb4-4cd8-99b5-523538908019","Type":"ContainerStarted","Data":"fe3ceb4da20823f6885f60cdfbcdcfdb52caf7c3d41f2f15efe745ab7ccb0c3a"} Oct 30 01:54:27.548254 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.548220 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" event={"ID":"63830843-4d31-45a7-83c9-9bb2248abe3b","Type":"ContainerStarted","Data":"0f2bd476b2115992a343a857c0baab5dbec6e4121a745c82ad3e0e24a24db26e"} Oct 30 01:54:27.548395 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.548277 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" event={"ID":"63830843-4d31-45a7-83c9-9bb2248abe3b","Type":"ContainerStarted","Data":"c3589b335c2f0748c19ef15df2c99dd3e03c09806aa654457efdbdf9cbdef5d6"} Oct 30 01:54:27.603119 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.600847 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" podStartSLOduration=1.600830819 podStartE2EDuration="1.600830819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:27.596457027 +0000 UTC m=+1497.175248109" watchObservedRunningTime="2026-10-30 01:54:27.600830819 +0000 UTC m=+1497.179621899" Oct 30 01:54:27.653365 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.648142 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" podStartSLOduration=1.648052187 podStartE2EDuration="1.648052187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:27.647980898 +0000 UTC m=+1497.226771979" watchObservedRunningTime="2026-10-30 01:54:27.648052187 +0000 UTC m=+1497.226843269" Oct 30 01:54:27.657290 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.657253 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:27.678419 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.678386 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:27.692037 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.692010 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62" Oct 30 01:54:27.770902 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.770849 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-maas-api-xks-ca\") pod \"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1\" (UID: \"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1\") " Oct 30 01:54:27.771084 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.770922 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-maas-api-openshift-ca\") pod \"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1\" (UID: \"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1\") " Oct 30 01:54:27.771084 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.770966 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-kube-api-access-2nld8\" (UniqueName: \"kubernetes.io/projected/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-kube-api-access-2nld8\") pod \"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1\" (UID: \"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1\") " Oct 30 01:54:27.772178 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.772129 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-maas-api-openshift-ca" pod "f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1" (UID: "f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:54:27.776299 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.776245 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-maas-api-xks-ca" pod "f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1" (UID: "f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:54:27.778191 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.778149 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-kube-api-access-2nld8" pod "f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1" (UID: "f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1"). InnerVolumeSpecName "kube-api-access-2nld8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:54:27.881113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.880223 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:54:27.881113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.880265 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:54:27.881113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.880281 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nld8\" (UniqueName: \"kubernetes.io/projected/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1-kube-api-access-2nld8\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:54:27.939115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:27.938957 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:28.573795 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:28.573767 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62" Oct 30 01:54:28.574137 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:28.573789 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62" event={"ID":"f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1","Type":"ContainerDied","Data":"47a474f3d4e2c55925f5c34b0a9ecd7ac4fe902d61a7bfea5a22fc3c6db88603"} Oct 30 01:54:28.574137 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:28.573835 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47a474f3d4e2c55925f5c34b0a9ecd7ac4fe902d61a7bfea5a22fc3c6db88603" Oct 30 01:54:28.661580 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:28.661214 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-sdrx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:54:28.661580 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:28.661284 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" podUID="6cd04c1d-6bb4-4cd8-99b5-523538908019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 01:54:28.680245 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:28.680191 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-kf6tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:54:28.680421 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:28.680269 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" podUID="63830843-4d31-45a7-83c9-9bb2248abe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 01:54:29.580471 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.580305 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t"] Oct 30 01:54:29.581226 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.581197 5998 cpu_manager.go:414] "containerMap: removing container" podUID="f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1" containerName="revoke-keys" Oct 30 01:54:29.581445 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.581412 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1" containerName="revoke-keys" Oct 30 01:54:29.581613 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.581600 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1" containerName="revoke-keys" Oct 30 01:54:29.585423 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.585401 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.603176 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.602438 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b93d51-openshift-default-dockercfg-qh2lt\"" Oct 30 01:54:29.637524 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.637488 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t"] Oct 30 01:54:29.669136 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.667969 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-sdrx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:54:29.669136 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.668027 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" podUID="6cd04c1d-6bb4-4cd8-99b5-523538908019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 01:54:29.679115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.678716 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-kf6tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:54:29.679115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.678785 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" podUID="63830843-4d31-45a7-83c9-9bb2248abe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 01:54:29.716114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.710132 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.716114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.710201 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.716114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.710238 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vpf9\" (UniqueName: \"kubernetes.io/projected/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-kube-api-access-4vpf9\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.718297 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.717355 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.718297 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.717469 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.718297 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.717678 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.718297 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.717708 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.718297 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.717788 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.718297 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.717870 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.820955 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.820908 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.821255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.821234 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.821404 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.821390 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.821547 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.821529 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.825119 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.821723 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.829219 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.829183 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.829412 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.829392 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vpf9\" (UniqueName: \"kubernetes.io/projected/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-kube-api-access-4vpf9\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.839157 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.835217 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.839407 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.839382 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.843110 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.830469 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.843297 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.830697 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.843397 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.824547 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.847107 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.843787 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.847274 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.829810 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.865500 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.862619 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.865500 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.862716 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.870708 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.869883 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.870859 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.870765 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vpf9\" (UniqueName: \"kubernetes.io/projected/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-kube-api-access-4vpf9\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.900115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.899515 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:29.978144 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:29.973192 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:30.358876 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:30.358846 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t"] Oct 30 01:54:30.360868 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:54:30.360841 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbcaa62_d63a_4bf4_ae72_8b091238cc2f.slice/crio-8498b798e9d19b08521afccdbe5b6a380b59304a63836ec7cf015089430a6bd6 WatchSource:0}: Error finding container 8498b798e9d19b08521afccdbe5b6a380b59304a63836ec7cf015089430a6bd6: Status 404 returned error can't find the container with id 8498b798e9d19b08521afccdbe5b6a380b59304a63836ec7cf015089430a6bd6 Oct 30 01:54:30.363498 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:30.363463 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:54:30.363605 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:30.363528 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:54:30.363605 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:30.363558 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:54:30.594198 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:30.594160 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" event={"ID":"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f","Type":"ContainerStarted","Data":"1f20376a1a3e8e4701c80f9584c3e1b1080554df0fa72ddaa99af2a76d5bf4dc"} Oct 30 01:54:30.594800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:30.594216 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" event={"ID":"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f","Type":"ContainerStarted","Data":"8498b798e9d19b08521afccdbe5b6a380b59304a63836ec7cf015089430a6bd6"} Oct 30 01:54:30.648171 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:30.643729 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" podStartSLOduration=1.643706714 podStartE2EDuration="1.643706714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:30.640606675 +0000 UTC m=+1500.219397773" watchObservedRunningTime="2026-10-30 01:54:30.643706714 +0000 UTC m=+1500.222497796" Oct 30 01:54:30.663115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:30.657489 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-sdrx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:54:30.663115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:30.657565 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" podUID="6cd04c1d-6bb4-4cd8-99b5-523538908019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 01:54:30.679361 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:30.679329 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-kf6tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:54:30.679551 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:30.679370 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" podUID="63830843-4d31-45a7-83c9-9bb2248abe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 01:54:30.920170 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:30.908186 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:31.690294 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:31.689573 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-sdrx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:54:31.690294 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:31.689633 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" podUID="6cd04c1d-6bb4-4cd8-99b5-523538908019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 01:54:31.690294 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:31.690084 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-kf6tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:54:31.690294 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:31.690152 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" podUID="63830843-4d31-45a7-83c9-9bb2248abe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 01:54:31.837350 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:31.837312 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:31.911328 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:31.911294 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:54:31.911606 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:31.911575 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" podUID="7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 01:54:32.657503 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:32.657464 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-sdrx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:54:32.657826 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:32.657793 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" podUID="6cd04c1d-6bb4-4cd8-99b5-523538908019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 01:54:32.679729 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:32.679499 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-kf6tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:54:32.679729 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:32.679599 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" podUID="63830843-4d31-45a7-83c9-9bb2248abe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 01:54:32.900523 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:32.900476 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:54:32.901216 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:32.901177 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" podUID="7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 01:54:33.505619 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:33.505577 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:33.657656 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:33.657614 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-sdrx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:54:33.657908 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:33.657685 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" podUID="6cd04c1d-6bb4-4cd8-99b5-523538908019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 01:54:33.679085 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:33.679053 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-kf6tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:54:33.679293 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:33.679133 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" podUID="63830843-4d31-45a7-83c9-9bb2248abe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 01:54:33.905177 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:33.904471 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:54:33.905177 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:33.904550 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" podUID="7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 01:54:34.527115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:34.521161 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:34.657505 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:34.657457 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-sdrx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 01:54:34.657689 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:34.657535 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" podUID="6cd04c1d-6bb4-4cd8-99b5-523538908019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 01:54:34.679532 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:34.679498 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-kf6tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:54:34.679691 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:34.679559 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" podUID="63830843-4d31-45a7-83c9-9bb2248abe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 30 01:54:34.905171 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:34.904526 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:54:34.905171 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:34.904603 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" podUID="7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 01:54:35.478680 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:35.478642 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:35.902518 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:35.902402 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:54:35.902780 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:35.902748 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" podUID="7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 01:54:36.682821 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:36.681125 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-kf6tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:54:36.682821 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:36.681206 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" podUID="63830843-4d31-45a7-83c9-9bb2248abe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:36.715275 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:36.715236 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-sdrx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:54:36.715587 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:36.715550 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" podUID="6cd04c1d-6bb4-4cd8-99b5-523538908019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:54:36.778970 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:36.778929 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:36.779763 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:36.779727 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:36.780741 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:36.780714 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 01:54:36.900901 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:36.900859 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:54:36.901148 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:36.900948 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" podUID="7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 01:54:36.933032 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:36.932964 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:36.933736 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:36.933717 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:36.940610 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:36.940584 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 01:54:37.694581 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:37.694527 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:37.850113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:37.849586 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:38.278627 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:38.278594 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:38.279045 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:38.279017 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:38.279857 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:38.279830 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 01:54:39.453149 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:39.453113 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:40.203492 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.203453 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8"] Oct 30 01:54:40.210536 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.210508 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.229113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.229069 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8"] Oct 30 01:54:40.237502 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.236772 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2933ff00\"/\"default-dockercfg-7xbts\"" Oct 30 01:54:40.237502 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.236859 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2933ff00\"/\"e2e-lifecycle-model-2933ff00-kserve-self-signed-certs\"" Oct 30 01:54:40.237710 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.237524 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2933ff00\"/\"kube-root-ca.crt\"" Oct 30 01:54:40.237710 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.237596 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2933ff00\"/\"openshift-service-ca.crt\"" Oct 30 01:54:40.247022 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.245050 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-tls-certs\") pod \"e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.247022 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.245139 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds4jl\" (UniqueName: \"kubernetes.io/projected/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-kube-api-access-ds4jl\") pod \"e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.247022 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.245186 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-dshm\") pod \"e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.247022 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.245301 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-model-cache\") pod \"e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.247022 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.245358 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-home\") pod \"e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.247022 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.245437 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-tmp-dir\") pod \"e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.250709 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.250681 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:40.346712 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.346655 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-model-cache\") pod \"e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.346712 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.346726 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-home\") pod \"e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.346973 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.346785 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-tmp-dir\") pod \"e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.346973 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.346856 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-tls-certs\") pod \"e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.346973 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.346910 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds4jl\" (UniqueName: \"kubernetes.io/projected/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-kube-api-access-ds4jl\") pod \"e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.346973 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.346946 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-dshm\") pod \"e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.347230 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.347198 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-model-cache\") pod \"e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.347622 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.347392 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-home\") pod \"e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.347763 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.347708 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-tmp-dir\") pod \"e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.349974 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.349945 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-dshm\") pod \"e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.350473 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.350456 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-tls-certs\") pod \"e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.369978 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.369946 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds4jl\" (UniqueName: \"kubernetes.io/projected/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-kube-api-access-ds4jl\") pod \"e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.528422 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.528327 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:40.878494 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:40.878469 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8"] Oct 30 01:54:40.881328 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:54:40.881298 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb036aa6_48d5_465a_a4c8_5860ce3d69eb.slice/crio-c80e32247fb93e3d164e18694fbaf5f22eb5500bad1666c7140ed895008a4cb1 WatchSource:0}: Error finding container c80e32247fb93e3d164e18694fbaf5f22eb5500bad1666c7140ed895008a4cb1: Status 404 returned error can't find the container with id c80e32247fb93e3d164e18694fbaf5f22eb5500bad1666c7140ed895008a4cb1 Oct 30 01:54:41.159766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.159678 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n"] Oct 30 01:54:41.164170 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.164120 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.207599 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.207388 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n"] Oct 30 01:54:41.264883 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.264848 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.265064 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.264889 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321d663c-9621-4376-9ef9-f8aeeafa3f20-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.265064 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.264965 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.265180 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.265067 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.265180 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.265129 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpmxv\" (UniqueName: \"kubernetes.io/projected/321d663c-9621-4376-9ef9-f8aeeafa3f20-kube-api-access-dpmxv\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.265258 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.265227 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.265309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.265292 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.265439 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.265417 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.265485 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.265450 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.309225 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.309175 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:41.366496 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.366446 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.366687 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.366513 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.366687 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.366574 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.366687 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.366604 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321d663c-9621-4376-9ef9-f8aeeafa3f20-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.366922 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.366711 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.366922 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.366829 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.366922 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.366879 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpmxv\" (UniqueName: \"kubernetes.io/projected/321d663c-9621-4376-9ef9-f8aeeafa3f20-kube-api-access-dpmxv\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.367081 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.367029 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.367232 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.367208 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.367547 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.367523 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.367679 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.367611 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.367757 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.367622 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.367757 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.367699 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321d663c-9621-4376-9ef9-f8aeeafa3f20-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.367890 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.367868 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.370288 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.370264 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.376502 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.376252 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.396329 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.396295 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.397525 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.397501 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpmxv\" (UniqueName: \"kubernetes.io/projected/321d663c-9621-4376-9ef9-f8aeeafa3f20-kube-api-access-dpmxv\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.479108 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.479057 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:41.809657 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.809626 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n"] Oct 30 01:54:41.809979 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:54:41.809767 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321d663c_9621_4376_9ef9_f8aeeafa3f20.slice/crio-01663ffeeaadb2063a5a1dbf81aa5940dbe1ce3e20badeac95e1b34291bb1db0 WatchSource:0}: Error finding container 01663ffeeaadb2063a5a1dbf81aa5940dbe1ce3e20badeac95e1b34291bb1db0: Status 404 returned error can't find the container with id 01663ffeeaadb2063a5a1dbf81aa5940dbe1ce3e20badeac95e1b34291bb1db0 Oct 30 01:54:41.812694 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.812653 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:54:41.812786 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.812744 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:54:41.812826 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.812788 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:54:41.867638 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.867608 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" event={"ID":"321d663c-9621-4376-9ef9-f8aeeafa3f20","Type":"ContainerStarted","Data":"01663ffeeaadb2063a5a1dbf81aa5940dbe1ce3e20badeac95e1b34291bb1db0"} Oct 30 01:54:41.869381 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.869346 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" event={"ID":"cb036aa6-48d5-465a-a4c8-5860ce3d69eb","Type":"ContainerStarted","Data":"e9b32b219086d8e69c0615c448d5e40b7e36d11fb0df3368ac70704e6cd8d4f3"} Oct 30 01:54:41.869553 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.869392 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" event={"ID":"cb036aa6-48d5-465a-a4c8-5860ce3d69eb","Type":"ContainerStarted","Data":"c80e32247fb93e3d164e18694fbaf5f22eb5500bad1666c7140ed895008a4cb1"} Oct 30 01:54:41.918220 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:41.918159 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" podStartSLOduration=1.918141342 podStartE2EDuration="1.918141342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:41.910356824 +0000 UTC m=+1511.489147905" watchObservedRunningTime="2026-10-30 01:54:41.918141342 +0000 UTC m=+1511.496932422" Oct 30 01:54:42.529453 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:42.529418 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:42.878143 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:42.878024 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" event={"ID":"321d663c-9621-4376-9ef9-f8aeeafa3f20","Type":"ContainerStarted","Data":"d1567a028e016acdee7e90f00c17b498b730c1e4ebcda5f7269d2ad1cb2bd597"} Oct 30 01:54:42.956268 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:42.955463 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" podStartSLOduration=1.955440606 podStartE2EDuration="1.955440606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:42.935573691 +0000 UTC m=+1512.514364774" watchObservedRunningTime="2026-10-30 01:54:42.955440606 +0000 UTC m=+1512.534231688" Oct 30 01:54:43.072112 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:43.072062 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:43.479887 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:43.479848 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:43.481212 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:43.481185 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:54:43.481363 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:43.481239 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" podUID="321d663c-9621-4376-9ef9-f8aeeafa3f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 01:54:43.946753 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:43.946713 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:44.479879 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.479845 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:54:44.480050 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.479897 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" podUID="321d663c-9621-4376-9ef9-f8aeeafa3f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 01:54:44.547743 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.546188 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w"] Oct 30 01:54:44.559680 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.556943 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w"] Oct 30 01:54:44.559680 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.557066 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.565475 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.564258 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:44.639113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.638262 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.639113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.638341 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3605446f-2664-4b96-9a9f-e13aab64faee-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.639113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.638495 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3605446f-2664-4b96-9a9f-e13aab64faee-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.639113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.638584 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxkwp\" (UniqueName: \"kubernetes.io/projected/3605446f-2664-4b96-9a9f-e13aab64faee-kube-api-access-hxkwp\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.639113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.638736 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.639113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.638798 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.639113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.638853 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3605446f-2664-4b96-9a9f-e13aab64faee-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.639113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.638899 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.639113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.638960 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.740853 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.740762 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.741020 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.740851 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.741020 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.740875 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3605446f-2664-4b96-9a9f-e13aab64faee-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.741020 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.740922 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3605446f-2664-4b96-9a9f-e13aab64faee-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.741020 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.740968 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxkwp\" (UniqueName: \"kubernetes.io/projected/3605446f-2664-4b96-9a9f-e13aab64faee-kube-api-access-hxkwp\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.741277 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.741044 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.741277 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.741084 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.741277 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.741137 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3605446f-2664-4b96-9a9f-e13aab64faee-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.741277 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.741181 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.741478 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.741344 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.741548 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.741525 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.741787 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.741763 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3605446f-2664-4b96-9a9f-e13aab64faee-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.741902 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.741880 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.742058 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.742036 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.743978 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.743911 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.744456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.744432 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3605446f-2664-4b96-9a9f-e13aab64faee-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.763849 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.763824 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3605446f-2664-4b96-9a9f-e13aab64faee-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.764192 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.764169 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxkwp\" (UniqueName: \"kubernetes.io/projected/3605446f-2664-4b96-9a9f-e13aab64faee-kube-api-access-hxkwp\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.874850 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.874807 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:44.897247 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:44.897213 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:45.194782 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:45.194641 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:45.395131 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:45.376352 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:45.491525 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:45.488057 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:54:45.491525 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:45.488136 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" podUID="321d663c-9621-4376-9ef9-f8aeeafa3f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 01:54:45.493265 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:45.493218 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w"] Oct 30 01:54:45.497891 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:45.497850 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:54:45.498015 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:45.497920 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:54:45.498015 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:45.497947 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:54:45.947113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:45.943676 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" event={"ID":"3605446f-2664-4b96-9a9f-e13aab64faee","Type":"ContainerStarted","Data":"883b0ca245b89825ab2c2b7bccd313bc9c11f9633acbaafebbaccfb62607a603"} Oct 30 01:54:45.947113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:45.943749 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" event={"ID":"3605446f-2664-4b96-9a9f-e13aab64faee","Type":"ContainerStarted","Data":"4f029ce0ddfbac192dc08b989105f68f1e167389c932199620d1764b2bd9004e"} Oct 30 01:54:46.056141 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:46.042622 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" podStartSLOduration=2.042601373 podStartE2EDuration="2.042601373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:46.023573548 +0000 UTC m=+1515.602364631" watchObservedRunningTime="2026-10-30 01:54:46.042601373 +0000 UTC m=+1515.621392455" Oct 30 01:54:46.482109 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:46.482009 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:54:46.482109 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:46.482073 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" podUID="321d663c-9621-4376-9ef9-f8aeeafa3f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 01:54:46.875129 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:46.875026 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:46.876464 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:46.876439 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:54:46.876666 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:46.876640 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" podUID="3605446f-2664-4b96-9a9f-e13aab64faee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 01:54:47.488115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:47.487256 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:54:47.488115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:47.487338 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" podUID="321d663c-9621-4376-9ef9-f8aeeafa3f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 01:54:47.843728 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:47.843637 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:47.877509 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:47.876959 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:54:47.877509 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:47.877016 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" podUID="3605446f-2664-4b96-9a9f-e13aab64faee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 01:54:48.479522 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:48.479481 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:54:48.482651 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:48.482613 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" podUID="321d663c-9621-4376-9ef9-f8aeeafa3f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 01:54:48.875668 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:48.875587 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:54:48.875668 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:48.875644 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" podUID="3605446f-2664-4b96-9a9f-e13aab64faee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 01:54:49.480116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:49.480057 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:54:49.480287 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:49.480131 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" podUID="321d663c-9621-4376-9ef9-f8aeeafa3f20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 01:54:49.570187 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:49.570154 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:49.750424 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:49.750322 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:49.876200 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:49.876159 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:54:49.876736 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:49.876232 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" podUID="3605446f-2664-4b96-9a9f-e13aab64faee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 01:54:50.484800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:50.484765 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:50.485241 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:50.485217 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:50.486128 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:50.486075 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 01:54:50.531113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:50.530544 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:50.545192 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:50.545165 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:50.665540 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:50.665487 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:50.876044 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:50.875966 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:54:50.876044 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:50.876022 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" podUID="3605446f-2664-4b96-9a9f-e13aab64faee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 01:54:51.079455 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:51.079422 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:54:51.875720 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:51.875654 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:54:51.875984 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:51.875740 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" podUID="3605446f-2664-4b96-9a9f-e13aab64faee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 01:54:51.939356 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:51.939312 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:52.061520 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:52.061462 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:52.876050 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:52.876013 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:54:52.876459 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:52.876082 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" podUID="3605446f-2664-4b96-9a9f-e13aab64faee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 01:54:52.949475 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:52.949445 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:53.134641 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:53.134551 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:53.879560 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:53.879523 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:53.880013 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:53.879943 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:53.880749 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:53.880728 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 01:54:55.183405 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:55.183376 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:55.226771 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:54:55.226700 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:54:56.741141 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:56.741102 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:58.344884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:58.344845 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:54:59.530777 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.530738 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8"] Oct 30 01:54:59.535329 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.535299 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.555282 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.555257 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2198e4-openshift-default-dockercfg-zhwrb\"" Oct 30 01:54:59.587255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.587218 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8"] Oct 30 01:54:59.686375 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.686336 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-credential-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.686555 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.686381 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-workload-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.686555 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.686462 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istiod-ca-cert\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.686555 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.686506 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhwsz\" (UniqueName: \"kubernetes.io/projected/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-kube-api-access-rhwsz\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.686667 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.686549 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-data\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.686667 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.686581 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-token\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.686667 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.686612 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-workload-certs\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.686755 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.686703 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-envoy\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.686755 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.686728 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-podinfo\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.787758 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.787678 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-workload-certs\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.787758 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.787742 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-envoy\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.787943 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.787786 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-podinfo\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.788010 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.787992 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-credential-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.788069 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.788034 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-workload-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.788166 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.788071 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istiod-ca-cert\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.788226 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.788166 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-workload-certs\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.788226 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.788187 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhwsz\" (UniqueName: \"kubernetes.io/projected/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-kube-api-access-rhwsz\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.788345 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.788234 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-data\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.788345 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.788266 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-token\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.788452 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.788388 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-credential-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.788508 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.788445 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-workload-socket\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.788915 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.788658 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-data\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.797115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.789434 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istiod-ca-cert\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.799153 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.799128 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-envoy\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.799629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.799599 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-podinfo\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.840616 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.840583 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-token\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.840870 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.840850 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhwsz\" (UniqueName: \"kubernetes.io/projected/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-kube-api-access-rhwsz\") pod \"e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:54:59.865213 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:54:59.865176 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:55:00.001076 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:00.001037 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:00.199261 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:00.199231 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8"] Oct 30 01:55:00.201493 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:55:00.201463 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2f69ea_77d6_4b18_bb09_a6ca90e7d51b.slice/crio-6eda81a081146deb40ea184cd72bfc26a9fd4d22b124b7dd82399fd16d6e0051 WatchSource:0}: Error finding container 6eda81a081146deb40ea184cd72bfc26a9fd4d22b124b7dd82399fd16d6e0051: Status 404 returned error can't find the container with id 6eda81a081146deb40ea184cd72bfc26a9fd4d22b124b7dd82399fd16d6e0051 Oct 30 01:55:00.204077 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:00.204043 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:55:00.204193 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:00.204137 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:55:00.204193 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:00.204183 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:55:01.165456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:01.165423 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" event={"ID":"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b","Type":"ContainerStarted","Data":"b95b07aef71884a8226ca6fa15f5289872fef618cbbc949bf88e5dd6ffb6300d"} Oct 30 01:55:01.166014 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:01.165987 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" event={"ID":"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b","Type":"ContainerStarted","Data":"6eda81a081146deb40ea184cd72bfc26a9fd4d22b124b7dd82399fd16d6e0051"} Oct 30 01:55:01.224116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:01.223741 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" podStartSLOduration=2.223724256 podStartE2EDuration="2.223724256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:01.222738596 +0000 UTC m=+1530.801529677" watchObservedRunningTime="2026-10-30 01:55:01.223724256 +0000 UTC m=+1530.802515339" Oct 30 01:55:01.865769 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:01.865732 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:55:01.867500 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:01.867473 5998 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:55:01.867636 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:01.867520 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" podUID="8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 01:55:01.967382 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:01.967348 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:02.865497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:02.865462 5998 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:55:02.865918 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:02.865516 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" podUID="8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 01:55:03.866402 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:03.866359 5998 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:55:03.866803 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:03.866436 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" podUID="8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 01:55:04.252753 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:04.252716 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:04.865731 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:04.865694 5998 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:55:04.865924 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:04.865753 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" podUID="8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 01:55:05.866434 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:05.866395 5998 patch_prober.go:29] interesting pod/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:55:05.866872 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:05.866468 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" podUID="8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 01:55:06.450353 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:06.450317 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:06.557605 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:06.557572 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:06.663869 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:06.663830 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:06.870309 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:06.870235 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:55:06.870755 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:06.870542 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:55:06.871370 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:06.871351 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:55:08.552325 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.552283 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:08.553072 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.553028 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq"] Oct 30 01:55:08.559243 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.559215 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq" Oct 30 01:55:08.573756 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.573726 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq"] Oct 30 01:55:08.580609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.580056 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:55:08.580609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.580079 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:55:08.580609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.580176 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:55:08.591124 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.590773 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:55:08.674682 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.674641 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcdgq\" (UniqueName: \"kubernetes.io/projected/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-kube-api-access-pcdgq\") pod \"maas-api-revoke-sub-e9a6c10e-dlfcq\" (UID: \"ed6d64f0-4eda-48ba-aa2c-954d0555eb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq" Oct 30 01:55:08.674879 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.674762 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9a6c10e-dlfcq\" (UID: \"ed6d64f0-4eda-48ba-aa2c-954d0555eb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq" Oct 30 01:55:08.674879 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.674789 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9a6c10e-dlfcq\" (UID: \"ed6d64f0-4eda-48ba-aa2c-954d0555eb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq" Oct 30 01:55:08.776376 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.776333 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcdgq\" (UniqueName: \"kubernetes.io/projected/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-kube-api-access-pcdgq\") pod \"maas-api-revoke-sub-e9a6c10e-dlfcq\" (UID: \"ed6d64f0-4eda-48ba-aa2c-954d0555eb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq" Oct 30 01:55:08.776563 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.776434 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9a6c10e-dlfcq\" (UID: \"ed6d64f0-4eda-48ba-aa2c-954d0555eb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq" Oct 30 01:55:08.776563 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.776457 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9a6c10e-dlfcq\" (UID: \"ed6d64f0-4eda-48ba-aa2c-954d0555eb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq" Oct 30 01:55:08.777181 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.777158 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9a6c10e-dlfcq\" (UID: \"ed6d64f0-4eda-48ba-aa2c-954d0555eb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq" Oct 30 01:55:08.779015 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.778991 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9a6c10e-dlfcq\" (UID: \"ed6d64f0-4eda-48ba-aa2c-954d0555eb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq" Oct 30 01:55:08.803250 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.803190 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcdgq\" (UniqueName: \"kubernetes.io/projected/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-kube-api-access-pcdgq\") pod \"maas-api-revoke-sub-e9a6c10e-dlfcq\" (UID: \"ed6d64f0-4eda-48ba-aa2c-954d0555eb83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq" Oct 30 01:55:08.875363 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.875322 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq" Oct 30 01:55:08.997350 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:08.995638 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:55:09.213073 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:09.213033 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq"] Oct 30 01:55:09.248881 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:09.248839 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq" event={"ID":"ed6d64f0-4eda-48ba-aa2c-954d0555eb83","Type":"ContainerStarted","Data":"8c4a94ead4b037c1679de3b4713e4e7ecbbccb1ae0c74a98a36409838c1ab714"} Oct 30 01:55:10.261349 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:10.261307 5998 generic.go:356] "Generic (PLEG): container finished" podID="ed6d64f0-4eda-48ba-aa2c-954d0555eb83" containerID="6760573b71328f8026a62594424f04a19961483c33ee2d87d11ab255021046e4" exitCode=0 Oct 30 01:55:10.261869 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:10.261390 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq" event={"ID":"ed6d64f0-4eda-48ba-aa2c-954d0555eb83","Type":"ContainerDied","Data":"6760573b71328f8026a62594424f04a19961483c33ee2d87d11ab255021046e4"} Oct 30 01:55:10.570925 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:10.570881 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:11.456775 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:11.456747 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq" Oct 30 01:55:11.515828 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:11.515794 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-kube-api-access-pcdgq\" (UniqueName: \"kubernetes.io/projected/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-kube-api-access-pcdgq\") pod \"ed6d64f0-4eda-48ba-aa2c-954d0555eb83\" (UID: \"ed6d64f0-4eda-48ba-aa2c-954d0555eb83\") " Oct 30 01:55:11.516046 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:11.515913 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-maas-api-openshift-ca\") pod \"ed6d64f0-4eda-48ba-aa2c-954d0555eb83\" (UID: \"ed6d64f0-4eda-48ba-aa2c-954d0555eb83\") " Oct 30 01:55:11.516046 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:11.515954 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-maas-api-xks-ca\") pod \"ed6d64f0-4eda-48ba-aa2c-954d0555eb83\" (UID: \"ed6d64f0-4eda-48ba-aa2c-954d0555eb83\") " Oct 30 01:55:11.516442 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:11.516413 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-maas-api-openshift-ca" pod "ed6d64f0-4eda-48ba-aa2c-954d0555eb83" (UID: "ed6d64f0-4eda-48ba-aa2c-954d0555eb83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:55:11.516798 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:11.516774 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:55:11.518730 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:11.518704 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-kube-api-access-pcdgq" pod "ed6d64f0-4eda-48ba-aa2c-954d0555eb83" (UID: "ed6d64f0-4eda-48ba-aa2c-954d0555eb83"). InnerVolumeSpecName "kube-api-access-pcdgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:11.519167 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:11.519138 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-maas-api-xks-ca" pod "ed6d64f0-4eda-48ba-aa2c-954d0555eb83" (UID: "ed6d64f0-4eda-48ba-aa2c-954d0555eb83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:55:11.618595 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:11.618559 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcdgq\" (UniqueName: \"kubernetes.io/projected/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-kube-api-access-pcdgq\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:55:11.618595 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:11.618591 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6d64f0-4eda-48ba-aa2c-954d0555eb83-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:55:12.228029 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:12.227982 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:12.300303 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:12.300249 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq" event={"ID":"ed6d64f0-4eda-48ba-aa2c-954d0555eb83","Type":"ContainerDied","Data":"8c4a94ead4b037c1679de3b4713e4e7ecbbccb1ae0c74a98a36409838c1ab714"} Oct 30 01:55:12.300567 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:12.300362 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c4a94ead4b037c1679de3b4713e4e7ecbbccb1ae0c74a98a36409838c1ab714" Oct 30 01:55:12.300567 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:12.300451 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq" Oct 30 01:55:13.892627 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:13.892589 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:15.475995 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:15.475960 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:17.132944 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:17.132906 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:18.798902 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:18.798845 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:18.909179 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:18.909144 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch"] Oct 30 01:55:18.909804 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:18.909788 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ed6d64f0-4eda-48ba-aa2c-954d0555eb83" containerName="revoke-keys" Oct 30 01:55:18.909870 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:18.909805 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6d64f0-4eda-48ba-aa2c-954d0555eb83" containerName="revoke-keys" Oct 30 01:55:18.909909 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:18.909897 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6d64f0-4eda-48ba-aa2c-954d0555eb83" containerName="revoke-keys" Oct 30 01:55:18.914770 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:18.914751 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch" Oct 30 01:55:18.930690 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:18.930666 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:55:18.941592 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:18.941563 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:55:18.941820 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:18.941579 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:55:18.941918 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:18.941667 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:55:18.947398 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:18.947360 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch"] Oct 30 01:55:19.019534 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:19.019504 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-649jq\" (UniqueName: \"kubernetes.io/projected/f95dafc6-6159-4e69-9ab1-1090d5b244d1-kube-api-access-649jq\") pod \"maas-api-revoke-sub-12e41f89-prsch\" (UID: \"f95dafc6-6159-4e69-9ab1-1090d5b244d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch" Oct 30 01:55:19.019742 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:19.019547 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95dafc6-6159-4e69-9ab1-1090d5b244d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e41f89-prsch\" (UID: \"f95dafc6-6159-4e69-9ab1-1090d5b244d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch" Oct 30 01:55:19.019742 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:19.019659 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95dafc6-6159-4e69-9ab1-1090d5b244d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e41f89-prsch\" (UID: \"f95dafc6-6159-4e69-9ab1-1090d5b244d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch" Oct 30 01:55:19.121389 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:19.121305 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-649jq\" (UniqueName: \"kubernetes.io/projected/f95dafc6-6159-4e69-9ab1-1090d5b244d1-kube-api-access-649jq\") pod \"maas-api-revoke-sub-12e41f89-prsch\" (UID: \"f95dafc6-6159-4e69-9ab1-1090d5b244d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch" Oct 30 01:55:19.121389 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:19.121356 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95dafc6-6159-4e69-9ab1-1090d5b244d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e41f89-prsch\" (UID: \"f95dafc6-6159-4e69-9ab1-1090d5b244d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch" Oct 30 01:55:19.121389 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:19.121386 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95dafc6-6159-4e69-9ab1-1090d5b244d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e41f89-prsch\" (UID: \"f95dafc6-6159-4e69-9ab1-1090d5b244d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch" Oct 30 01:55:19.122164 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:19.122138 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95dafc6-6159-4e69-9ab1-1090d5b244d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12e41f89-prsch\" (UID: \"f95dafc6-6159-4e69-9ab1-1090d5b244d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch" Oct 30 01:55:19.123786 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:19.123764 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95dafc6-6159-4e69-9ab1-1090d5b244d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12e41f89-prsch\" (UID: \"f95dafc6-6159-4e69-9ab1-1090d5b244d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch" Oct 30 01:55:19.144227 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:19.144195 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-649jq\" (UniqueName: \"kubernetes.io/projected/f95dafc6-6159-4e69-9ab1-1090d5b244d1-kube-api-access-649jq\") pod \"maas-api-revoke-sub-12e41f89-prsch\" (UID: \"f95dafc6-6159-4e69-9ab1-1090d5b244d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch" Oct 30 01:55:19.230958 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:19.230921 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch" Oct 30 01:55:19.551820 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:19.551789 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch"] Oct 30 01:55:19.561032 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:55:19.560998 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95dafc6_6159_4e69_9ab1_1090d5b244d1.slice/crio-032b81e9da198b74f0b67fb5c073a3326c90b04d2299b61187f9dbf235537b27 WatchSource:0}: Error finding container 032b81e9da198b74f0b67fb5c073a3326c90b04d2299b61187f9dbf235537b27: Status 404 returned error can't find the container with id 032b81e9da198b74f0b67fb5c073a3326c90b04d2299b61187f9dbf235537b27 Oct 30 01:55:20.369608 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:20.369568 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch" event={"ID":"f95dafc6-6159-4e69-9ab1-1090d5b244d1","Type":"ContainerStarted","Data":"9d67d9deb181e72c75fe8aeca76f9b04c9ef180b888541418e05ae395271ef79"} Oct 30 01:55:20.369990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:20.369638 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch" event={"ID":"f95dafc6-6159-4e69-9ab1-1090d5b244d1","Type":"ContainerStarted","Data":"032b81e9da198b74f0b67fb5c073a3326c90b04d2299b61187f9dbf235537b27"} Oct 30 01:55:20.410700 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:20.410629 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch" podStartSLOduration=2.410611009 podStartE2EDuration="2.410611009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:20.409318317 +0000 UTC m=+1549.988109400" watchObservedRunningTime="2026-10-30 01:55:20.410611009 +0000 UTC m=+1549.989402091" Oct 30 01:55:20.592106 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:20.592056 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:21.381587 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:21.381552 5998 generic.go:356] "Generic (PLEG): container finished" podID="f95dafc6-6159-4e69-9ab1-1090d5b244d1" containerID="9d67d9deb181e72c75fe8aeca76f9b04c9ef180b888541418e05ae395271ef79" exitCode=0 Oct 30 01:55:21.382046 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:21.381644 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch" event={"ID":"f95dafc6-6159-4e69-9ab1-1090d5b244d1","Type":"ContainerDied","Data":"9d67d9deb181e72c75fe8aeca76f9b04c9ef180b888541418e05ae395271ef79"} Oct 30 01:55:22.171355 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:22.171325 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:22.561051 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:22.561026 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch" Oct 30 01:55:22.669389 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:22.669354 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95dafc6-6159-4e69-9ab1-1090d5b244d1-kube-api-access-649jq\" (UniqueName: \"kubernetes.io/projected/f95dafc6-6159-4e69-9ab1-1090d5b244d1-kube-api-access-649jq\") pod \"f95dafc6-6159-4e69-9ab1-1090d5b244d1\" (UID: \"f95dafc6-6159-4e69-9ab1-1090d5b244d1\") " Oct 30 01:55:22.669595 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:22.669408 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95dafc6-6159-4e69-9ab1-1090d5b244d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95dafc6-6159-4e69-9ab1-1090d5b244d1-maas-api-openshift-ca\") pod \"f95dafc6-6159-4e69-9ab1-1090d5b244d1\" (UID: \"f95dafc6-6159-4e69-9ab1-1090d5b244d1\") " Oct 30 01:55:22.669595 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:22.669480 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f95dafc6-6159-4e69-9ab1-1090d5b244d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95dafc6-6159-4e69-9ab1-1090d5b244d1-maas-api-xks-ca\") pod \"f95dafc6-6159-4e69-9ab1-1090d5b244d1\" (UID: \"f95dafc6-6159-4e69-9ab1-1090d5b244d1\") " Oct 30 01:55:22.669942 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:22.669841 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95dafc6-6159-4e69-9ab1-1090d5b244d1-maas-api-openshift-ca" pod "f95dafc6-6159-4e69-9ab1-1090d5b244d1" (UID: "f95dafc6-6159-4e69-9ab1-1090d5b244d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:55:22.670383 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:22.670316 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f95dafc6-6159-4e69-9ab1-1090d5b244d1-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:55:22.671826 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:22.671799 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95dafc6-6159-4e69-9ab1-1090d5b244d1-kube-api-access-649jq" pod "f95dafc6-6159-4e69-9ab1-1090d5b244d1" (UID: "f95dafc6-6159-4e69-9ab1-1090d5b244d1"). InnerVolumeSpecName "kube-api-access-649jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:22.671954 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:22.671890 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f95dafc6-6159-4e69-9ab1-1090d5b244d1-maas-api-xks-ca" pod "f95dafc6-6159-4e69-9ab1-1090d5b244d1" (UID: "f95dafc6-6159-4e69-9ab1-1090d5b244d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:55:22.772154 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:22.772038 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-649jq\" (UniqueName: \"kubernetes.io/projected/f95dafc6-6159-4e69-9ab1-1090d5b244d1-kube-api-access-649jq\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:55:22.772154 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:22.772073 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f95dafc6-6159-4e69-9ab1-1090d5b244d1-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:55:23.397028 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:23.396994 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch" Oct 30 01:55:23.397244 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:23.397031 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch" event={"ID":"f95dafc6-6159-4e69-9ab1-1090d5b244d1","Type":"ContainerDied","Data":"032b81e9da198b74f0b67fb5c073a3326c90b04d2299b61187f9dbf235537b27"} Oct 30 01:55:23.397244 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:23.397107 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="032b81e9da198b74f0b67fb5c073a3326c90b04d2299b61187f9dbf235537b27" Oct 30 01:55:23.819064 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:23.819020 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:24.833796 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:24.833762 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:25.219772 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:55:25.219734 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:55:25.662060 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:25.662028 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:26.510315 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:26.510272 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:27.207538 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:27.207503 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:29.477630 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:29.477589 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:30.880565 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:30.880534 5998 scope.go:122] "RemoveContainer" containerID="c76d3be1c8b1c0aa31c76d7b6c72482d2804d1517ae1a09ead38eb78f2fd1f8e" Oct 30 01:55:30.891264 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:30.891238 5998 scope.go:122] "RemoveContainer" containerID="0d49e19040689eb87933d15c4e47dba469c1f26905f0459607ccd3c9915921e7" Oct 30 01:55:30.900513 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:30.900492 5998 scope.go:122] "RemoveContainer" containerID="e207db18a1c3f4b2d81ee1425af7812dee59380ff21f1ad169da49df339de5e7" Oct 30 01:55:30.910182 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:30.910154 5998 scope.go:122] "RemoveContainer" containerID="08dcfdb614c82903cf9eeb553e22ccb173de6399a2d65c7b6fe247bb92d1192d" Oct 30 01:55:31.308892 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.308850 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:31.416424 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.416147 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:31.637352 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.637270 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc"] Oct 30 01:55:31.638173 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.638151 5998 cpu_manager.go:414] "containerMap: removing container" podUID="f95dafc6-6159-4e69-9ab1-1090d5b244d1" containerName="revoke-keys" Oct 30 01:55:31.638273 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.638174 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95dafc6-6159-4e69-9ab1-1090d5b244d1" containerName="revoke-keys" Oct 30 01:55:31.638370 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.638356 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95dafc6-6159-4e69-9ab1-1090d5b244d1" containerName="revoke-keys" Oct 30 01:55:31.644482 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.644458 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc" Oct 30 01:55:31.659464 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.659439 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc"] Oct 30 01:55:31.667418 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.667331 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:55:31.667418 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.667345 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:55:31.667418 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.667392 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:55:31.672730 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.672708 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:55:31.722736 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.722694 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2bbe2c6-62hnc\" (UID: \"59c20a06-9416-4e1d-91d4-7eb88d1f6d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc" Oct 30 01:55:31.722962 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.722944 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spg7j\" (UniqueName: \"kubernetes.io/projected/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-kube-api-access-spg7j\") pod \"maas-api-revoke-sub-e2bbe2c6-62hnc\" (UID: \"59c20a06-9416-4e1d-91d4-7eb88d1f6d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc" Oct 30 01:55:31.723041 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.722970 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2bbe2c6-62hnc\" (UID: \"59c20a06-9416-4e1d-91d4-7eb88d1f6d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc" Oct 30 01:55:31.772341 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.771844 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf"] Oct 30 01:55:31.787374 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.787338 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:31.788753 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.788720 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf"] Oct 30 01:55:31.804723 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.803545 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-922b2c-openshift-default-dockercfg-p78gv\"" Oct 30 01:55:31.825057 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.825021 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spg7j\" (UniqueName: \"kubernetes.io/projected/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-kube-api-access-spg7j\") pod \"maas-api-revoke-sub-e2bbe2c6-62hnc\" (UID: \"59c20a06-9416-4e1d-91d4-7eb88d1f6d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc" Oct 30 01:55:31.825287 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.825067 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2bbe2c6-62hnc\" (UID: \"59c20a06-9416-4e1d-91d4-7eb88d1f6d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc" Oct 30 01:55:31.825287 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.825159 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2bbe2c6-62hnc\" (UID: \"59c20a06-9416-4e1d-91d4-7eb88d1f6d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc" Oct 30 01:55:31.825952 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.825930 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2bbe2c6-62hnc\" (UID: \"59c20a06-9416-4e1d-91d4-7eb88d1f6d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc" Oct 30 01:55:31.827783 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.827760 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:31.847744 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.843239 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2bbe2c6-62hnc\" (UID: \"59c20a06-9416-4e1d-91d4-7eb88d1f6d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc" Oct 30 01:55:31.859479 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.859453 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spg7j\" (UniqueName: \"kubernetes.io/projected/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-kube-api-access-spg7j\") pod \"maas-api-revoke-sub-e2bbe2c6-62hnc\" (UID: \"59c20a06-9416-4e1d-91d4-7eb88d1f6d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc" Oct 30 01:55:31.952125 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.939936 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d11c0bb-7924-485b-8514-1213bc06afca-istiod-ca-cert\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:31.952125 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.939999 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-credential-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:31.952125 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.940037 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-istio-data\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:31.952125 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.940067 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2bc7\" (UniqueName: \"kubernetes.io/projected/9d11c0bb-7924-485b-8514-1213bc06afca-kube-api-access-k2bc7\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:31.952125 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.940196 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-workload-certs\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:31.952125 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.940254 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d11c0bb-7924-485b-8514-1213bc06afca-istio-podinfo\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:31.952125 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.940317 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-workload-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:31.952125 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.940357 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-istio-envoy\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:31.952125 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.940452 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d11c0bb-7924-485b-8514-1213bc06afca-istio-token\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:31.988133 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:31.981673 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc" Oct 30 01:55:32.054127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.048626 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d11c0bb-7924-485b-8514-1213bc06afca-istio-token\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.054127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.048801 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d11c0bb-7924-485b-8514-1213bc06afca-istiod-ca-cert\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.054127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.048852 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-credential-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.054127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.048887 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-istio-data\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.054127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.048911 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2bc7\" (UniqueName: \"kubernetes.io/projected/9d11c0bb-7924-485b-8514-1213bc06afca-kube-api-access-k2bc7\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.054127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.049016 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-workload-certs\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.054127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.049072 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d11c0bb-7924-485b-8514-1213bc06afca-istio-podinfo\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.054127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.049181 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-workload-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.054127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.049221 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-istio-envoy\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.054127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.049518 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-credential-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.054127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.050544 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d11c0bb-7924-485b-8514-1213bc06afca-istiod-ca-cert\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.054127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.050796 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-workload-certs\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.054127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.051011 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-istio-data\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.072609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.055261 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-workload-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.072609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.056969 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-istio-envoy\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.092604 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.089732 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2bc7\" (UniqueName: \"kubernetes.io/projected/9d11c0bb-7924-485b-8514-1213bc06afca-kube-api-access-k2bc7\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.092604 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.090446 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d11c0bb-7924-485b-8514-1213bc06afca-istio-podinfo\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.092604 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.092545 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d11c0bb-7924-485b-8514-1213bc06afca-istio-token\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.112633 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.108233 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:32.112633 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.112337 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:32.447917 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.447880 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc"] Oct 30 01:55:32.452827 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:55:32.452796 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c20a06_9416_4e1d_91d4_7eb88d1f6d98.slice/crio-412c64fd6cb6d56e02e33e2c6092e3fa8566ed91964faa5786551d2cc3bb939b WatchSource:0}: Error finding container 412c64fd6cb6d56e02e33e2c6092e3fa8566ed91964faa5786551d2cc3bb939b: Status 404 returned error can't find the container with id 412c64fd6cb6d56e02e33e2c6092e3fa8566ed91964faa5786551d2cc3bb939b Oct 30 01:55:32.480251 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.480219 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc" event={"ID":"59c20a06-9416-4e1d-91d4-7eb88d1f6d98","Type":"ContainerStarted","Data":"412c64fd6cb6d56e02e33e2c6092e3fa8566ed91964faa5786551d2cc3bb939b"} Oct 30 01:55:32.513537 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.513508 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf"] Oct 30 01:55:32.515216 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:55:32.515176 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d11c0bb_7924_485b_8514_1213bc06afca.slice/crio-a4f906c22bafc1226dd709e86a6b1c4408aa0934a5793e68e36f9bdce96f1eca WatchSource:0}: Error finding container a4f906c22bafc1226dd709e86a6b1c4408aa0934a5793e68e36f9bdce96f1eca: Status 404 returned error can't find the container with id a4f906c22bafc1226dd709e86a6b1c4408aa0934a5793e68e36f9bdce96f1eca Oct 30 01:55:32.518702 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.518665 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:55:32.518787 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.518754 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:55:32.518830 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.518797 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:55:32.951429 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:32.937468 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:33.062623 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:33.062557 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:33.490730 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:33.490692 5998 generic.go:356] "Generic (PLEG): container finished" podID="59c20a06-9416-4e1d-91d4-7eb88d1f6d98" containerID="9890fbe8a63995ce74d5ee9f7b1147960f47f0579c4154f77fb7675706719b0c" exitCode=0 Oct 30 01:55:33.490947 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:33.490773 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc" event={"ID":"59c20a06-9416-4e1d-91d4-7eb88d1f6d98","Type":"ContainerDied","Data":"9890fbe8a63995ce74d5ee9f7b1147960f47f0579c4154f77fb7675706719b0c"} Oct 30 01:55:33.550780 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:33.549526 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" event={"ID":"9d11c0bb-7924-485b-8514-1213bc06afca","Type":"ContainerStarted","Data":"1ff026f8f2c1777effacccd9fef323c9c2ccd7c3b93bf51c04a605ff21594ed9"} Oct 30 01:55:33.550780 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:33.549597 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" event={"ID":"9d11c0bb-7924-485b-8514-1213bc06afca","Type":"ContainerStarted","Data":"a4f906c22bafc1226dd709e86a6b1c4408aa0934a5793e68e36f9bdce96f1eca"} Oct 30 01:55:33.628223 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:33.625142 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" podStartSLOduration=2.625074494 podStartE2EDuration="2.625074494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:33.606787438 +0000 UTC m=+1563.185578520" watchObservedRunningTime="2026-10-30 01:55:33.625074494 +0000 UTC m=+1563.203865578" Oct 30 01:55:33.667437 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:33.667372 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:34.115884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:34.115847 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:34.121666 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:34.121635 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:34.272566 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:34.272535 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:34.558236 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:34.557724 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:34.558609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:34.558587 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:55:34.647138 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:34.630822 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:34.760101 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:34.760062 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc" Oct 30 01:55:34.891908 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:34.891820 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-kube-api-access-spg7j\" (UniqueName: \"kubernetes.io/projected/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-kube-api-access-spg7j\") pod \"59c20a06-9416-4e1d-91d4-7eb88d1f6d98\" (UID: \"59c20a06-9416-4e1d-91d4-7eb88d1f6d98\") " Oct 30 01:55:34.892080 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:34.892065 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-maas-api-xks-ca\") pod \"59c20a06-9416-4e1d-91d4-7eb88d1f6d98\" (UID: \"59c20a06-9416-4e1d-91d4-7eb88d1f6d98\") " Oct 30 01:55:34.892165 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:34.892134 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-maas-api-openshift-ca\") pod \"59c20a06-9416-4e1d-91d4-7eb88d1f6d98\" (UID: \"59c20a06-9416-4e1d-91d4-7eb88d1f6d98\") " Oct 30 01:55:34.892517 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:34.892488 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-maas-api-openshift-ca" pod "59c20a06-9416-4e1d-91d4-7eb88d1f6d98" (UID: "59c20a06-9416-4e1d-91d4-7eb88d1f6d98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:55:34.892685 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:34.892670 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:55:34.894129 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:34.894065 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-kube-api-access-spg7j" pod "59c20a06-9416-4e1d-91d4-7eb88d1f6d98" (UID: "59c20a06-9416-4e1d-91d4-7eb88d1f6d98"). InnerVolumeSpecName "kube-api-access-spg7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:34.894129 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:34.894067 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-maas-api-xks-ca" pod "59c20a06-9416-4e1d-91d4-7eb88d1f6d98" (UID: "59c20a06-9416-4e1d-91d4-7eb88d1f6d98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:55:34.993855 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:34.993821 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:55:34.993855 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:34.993854 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spg7j\" (UniqueName: \"kubernetes.io/projected/59c20a06-9416-4e1d-91d4-7eb88d1f6d98-kube-api-access-spg7j\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:55:35.566311 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:35.566285 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc" Oct 30 01:55:35.566740 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:35.566314 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc" event={"ID":"59c20a06-9416-4e1d-91d4-7eb88d1f6d98","Type":"ContainerDied","Data":"412c64fd6cb6d56e02e33e2c6092e3fa8566ed91964faa5786551d2cc3bb939b"} Oct 30 01:55:35.566740 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:35.566362 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="412c64fd6cb6d56e02e33e2c6092e3fa8566ed91964faa5786551d2cc3bb939b" Oct 30 01:55:35.667010 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:35.666964 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:36.064701 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:36.064659 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k"] Oct 30 01:55:36.076789 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:36.076758 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80ca7ecc-p9c2k"] Oct 30 01:55:36.110959 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:36.110924 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:36.276070 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:36.276033 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:36.539887 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:36.539846 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dbdeeef-b6cd-43a2-b969-458bb5cbc302" path="/var/lib/kubelet/pods/7dbdeeef-b6cd-43a2-b969-458bb5cbc302/volumes" Oct 30 01:55:37.636635 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:37.636598 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:37.907714 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:37.907561 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:38.020287 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:38.020258 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:39.604237 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:39.604199 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:41.067253 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:41.067215 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:41.312859 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:41.312822 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:41.936557 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:41.936518 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:43.124424 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:43.124383 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:44.569407 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.569366 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm"] Oct 30 01:55:44.571957 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.571924 5998 cpu_manager.go:414] "containerMap: removing container" podUID="59c20a06-9416-4e1d-91d4-7eb88d1f6d98" containerName="revoke-keys" Oct 30 01:55:44.572262 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.572244 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c20a06-9416-4e1d-91d4-7eb88d1f6d98" containerName="revoke-keys" Oct 30 01:55:44.572882 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.572853 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c20a06-9416-4e1d-91d4-7eb88d1f6d98" containerName="revoke-keys" Oct 30 01:55:44.576289 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.576255 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:44.582780 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.580344 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.614060 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.613939 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm"] Oct 30 01:55:44.617562 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.617533 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn"] Oct 30 01:55:44.627143 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.623778 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.665339 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.657991 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn"] Oct 30 01:55:44.684161 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.684125 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.684334 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.684189 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.684334 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.684309 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.684476 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.684460 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.684523 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.684487 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.684523 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.684515 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.684629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.684578 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.685211 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.685189 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.685332 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.685239 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.685332 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.685283 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.685332 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.685320 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.685490 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.685349 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.685490 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.685379 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.685490 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.685402 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.685490 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.685446 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwlkn\" (UniqueName: \"kubernetes.io/projected/0b6f67a0-2dae-400a-abd7-7c65fbdac225-kube-api-access-cwlkn\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.685681 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.685502 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.685681 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.685540 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpjzt\" (UniqueName: \"kubernetes.io/projected/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-kube-api-access-qpjzt\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.685681 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.685564 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.795440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.786948 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.795440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.787009 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpjzt\" (UniqueName: \"kubernetes.io/projected/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-kube-api-access-qpjzt\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.795440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.787038 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.795440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.787164 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.795440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.787195 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.795440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.787243 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.795440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.787353 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.795440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.787375 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.795440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.787402 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.795440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.787425 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.795440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.787456 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.795440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.787493 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.795440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.787530 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.795440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.787567 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.795440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.787591 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.795440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.787617 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.795440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.787638 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.795440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.787680 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwlkn\" (UniqueName: \"kubernetes.io/projected/0b6f67a0-2dae-400a-abd7-7c65fbdac225-kube-api-access-cwlkn\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.797867 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.797212 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.806513 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.798820 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.814629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.807076 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.814629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.807412 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.814629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.807492 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.814629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.807651 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.814629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.810811 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.821470 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.816296 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.821993 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.821932 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.858126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.832721 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.858126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.835856 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.858126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.836359 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.858126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.836901 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwlkn\" (UniqueName: \"kubernetes.io/projected/0b6f67a0-2dae-400a-abd7-7c65fbdac225-kube-api-access-cwlkn\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.858126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.837497 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.858126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.851522 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.858126 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.853915 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.872270 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.861777 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpjzt\" (UniqueName: \"kubernetes.io/projected/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-kube-api-access-qpjzt\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.872270 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.867179 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:44.915262 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.914886 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:44.961296 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:44.961257 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:45.311646 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:45.311618 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm"] Oct 30 01:55:45.313191 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:55:45.313164 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6f67a0_2dae_400a_abd7_7c65fbdac225.slice/crio-08470e6ee30bb2d68b34d7d0481ec81555890caeb3f706d2740b4ce6f367f192 WatchSource:0}: Error finding container 08470e6ee30bb2d68b34d7d0481ec81555890caeb3f706d2740b4ce6f367f192: Status 404 returned error can't find the container with id 08470e6ee30bb2d68b34d7d0481ec81555890caeb3f706d2740b4ce6f367f192 Oct 30 01:55:45.315810 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:45.315774 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:55:45.315931 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:45.315841 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:55:45.315931 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:45.315871 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:55:45.321555 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:45.321528 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn"] Oct 30 01:55:45.329943 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:55:45.329910 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c105b9_9c67_48ab_821c_b16c9f9ef5e5.slice/crio-b449ac37823b6e933bf6bde8809b0caae18acd134843e4c6b8e0d8734f1b27da WatchSource:0}: Error finding container b449ac37823b6e933bf6bde8809b0caae18acd134843e4c6b8e0d8734f1b27da: Status 404 returned error can't find the container with id b449ac37823b6e933bf6bde8809b0caae18acd134843e4c6b8e0d8734f1b27da Oct 30 01:55:45.334189 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:45.334154 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:55:45.334298 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:45.334246 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:55:45.334298 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:45.334279 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:55:45.620074 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:45.619969 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:45.686387 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:45.686340 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" event={"ID":"0b6f67a0-2dae-400a-abd7-7c65fbdac225","Type":"ContainerStarted","Data":"3d85368bb06bb388c8f248f640ccf045c6d5eed581fad5fbf75b276628ea0ba0"} Oct 30 01:55:45.686595 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:45.686421 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" event={"ID":"0b6f67a0-2dae-400a-abd7-7c65fbdac225","Type":"ContainerStarted","Data":"08470e6ee30bb2d68b34d7d0481ec81555890caeb3f706d2740b4ce6f367f192"} Oct 30 01:55:45.694148 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:45.692167 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" event={"ID":"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5","Type":"ContainerStarted","Data":"aef65e1106b7ffb44a2e604aa32084565c216447430dc8fd93258f188c4b23a9"} Oct 30 01:55:45.694148 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:45.692235 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" event={"ID":"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5","Type":"ContainerStarted","Data":"b449ac37823b6e933bf6bde8809b0caae18acd134843e4c6b8e0d8734f1b27da"} Oct 30 01:55:45.735174 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:45.735110 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" podStartSLOduration=1.7350740930000002 podStartE2EDuration="1.735074093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:45.734589863 +0000 UTC m=+1575.313380945" watchObservedRunningTime="2026-10-30 01:55:45.735074093 +0000 UTC m=+1575.313865175" Oct 30 01:55:45.784593 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:45.784538 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" podStartSLOduration=1.784520715 podStartE2EDuration="1.784520715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:45.781274826 +0000 UTC m=+1575.360065908" watchObservedRunningTime="2026-10-30 01:55:45.784520715 +0000 UTC m=+1575.363311796" Oct 30 01:55:45.916057 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:45.916020 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:45.961820 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:45.961785 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:46.927578 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:46.927223 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:55:46.927578 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:46.927284 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" podUID="0b6f67a0-2dae-400a-abd7-7c65fbdac225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 01:55:46.964573 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:46.964526 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:55:46.964907 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:46.964820 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" podUID="a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 01:55:47.447119 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:47.446599 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:47.948118 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:47.941466 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:55:47.948118 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:47.941538 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" podUID="0b6f67a0-2dae-400a-abd7-7c65fbdac225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 01:55:47.975015 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:47.974845 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:55:47.975015 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:47.974916 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" podUID="a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 01:55:48.264102 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:48.264003 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:55:48.916513 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:48.916472 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:55:48.916856 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:48.916823 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" podUID="0b6f67a0-2dae-400a-abd7-7c65fbdac225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 01:55:48.961669 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:48.961627 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:55:48.962038 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:48.961687 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" podUID="a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 01:55:49.072161 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:49.072084 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r"] Oct 30 01:55:49.077104 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:49.077062 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b07babfb-9z79r"] Oct 30 01:55:49.364110 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:49.363997 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:49.876865 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:49.876795 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:49.917540 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:49.917497 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:55:49.917830 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:49.917796 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" podUID="0b6f67a0-2dae-400a-abd7-7c65fbdac225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 01:55:49.966753 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:49.966714 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:55:49.967381 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:49.967342 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" podUID="a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 01:55:50.034070 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:50.034019 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:50.595798 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:50.595759 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3013c307-263e-49ae-b79d-573925f7e85c" path="/var/lib/kubelet/pods/3013c307-263e-49ae-b79d-573925f7e85c/volumes" Oct 30 01:55:50.963177 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:50.936207 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Oct 30 01:55:50.980116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:50.964622 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:55:51.040155 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.037499 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" podUID="a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 01:55:51.040155 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.011191 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:51.040155 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:50.996329 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded" Oct 30 01:55:51.040155 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:50.996570 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:55:51.040155 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.037953 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" podUID="0b6f67a0-2dae-400a-abd7-7c65fbdac225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 01:55:51.043293 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.011226 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 01:55:51.060732 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.060693 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:51.076276 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.076232 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" podUID="cb036aa6-48d5-465a-a4c8-5860ce3d69eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:51.077277 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.076530 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:51.079453 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.076888 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:51.092923 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.091371 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:51.105221 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.105176 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 01:55:51.131252 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.131198 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:51.153213 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.153166 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:51.242643 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.242184 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:51.372494 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.372458 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:51.865114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.861941 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:51.933225 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.933188 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:55:51.933539 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.933488 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" podUID="0b6f67a0-2dae-400a-abd7-7c65fbdac225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 01:55:51.982276 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.981741 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:55:51.982276 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:51.981801 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" podUID="a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 01:55:52.043680 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.043633 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:52.048212 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.048152 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:52.055009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.054976 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:55:52.055294 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.055277 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:55:52.106205 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.085171 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:52.106205 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.085507 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:55:52.106205 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.092773 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:52.106205 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.103284 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:55:52.106205 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.103925 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:52.108223 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.108195 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:55:52.121175 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.109116 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:52.173131 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.135079 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:55:52.173131 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.136018 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 01:55:52.173131 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.139004 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:55:52.173131 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.158616 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:52.173131 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.158989 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:55:52.330629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.307401 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8"] Oct 30 01:55:52.330629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.317337 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" podUID="8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" containerName="istio-proxy" containerID="cri-o://b95b07aef71884a8226ca6fa15f5289872fef618cbbc949bf88e5dd6ffb6300d" gracePeriod=30 Oct 30 01:55:52.735114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.709377 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:52.780948 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:52.780735 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:53.054174 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.029337 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:55:53.084113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.083383 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" podUID="0b6f67a0-2dae-400a-abd7-7c65fbdac225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 01:55:53.084113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.059524 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:53.084113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.048360 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:53.084113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.036967 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-sdrx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:55:53.084113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.083651 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" podUID="6cd04c1d-6bb4-4cd8-99b5-523538908019" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:53.084113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.057454 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-kf6tq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:55:53.084113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.083714 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" podUID="63830843-4d31-45a7-83c9-9bb2248abe3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:53.084113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.060034 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:55:53.084113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.060669 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded" Oct 30 01:55:53.117459 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.085778 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" podUID="cb036aa6-48d5-465a-a4c8-5860ce3d69eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:53.117459 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.083773 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" podUID="a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 01:55:53.172603 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.170163 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:53.172896 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.172859 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 01:55:53.204516 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.173028 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:53.205018 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.178752 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:53.211168 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.183186 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:53.264249 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.264196 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 01:55:53.341714 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.341602 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:53.628801 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.628648 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded" Oct 30 01:55:53.796116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.774731 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:53.986117 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.922879 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:55:53.986584 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.986530 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" podUID="0b6f67a0-2dae-400a-abd7-7c65fbdac225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 01:55:54.016255 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:53.998309 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:55:54.050115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.016505 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" podUID="a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 01:55:54.051506 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.025559 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:54.103148 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.095214 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:54.103148 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.095621 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 01:55:54.103148 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.096761 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" podUID="cb036aa6-48d5-465a-a4c8-5860ce3d69eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:54.103148 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.097192 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:55:54.117125 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.116737 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" podUID="f02332d2-b193-4f97-bb47-d0d0904bdc9a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.27:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:54.176135 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.174878 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 01:55:54.248253 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.230660 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:54.248253 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.231024 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:54.248253 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.232128 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:54.248253 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.232367 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 01:55:54.352726 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.326165 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 01:55:54.352726 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.326603 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:55:54.352726 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.351339 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 01:55:54.435139 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.435083 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:54.573386 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.477510 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8pws6" podUID="f02332d2-b193-4f97-bb47-d0d0904bdc9a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.27:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:54.865739 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.865658 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 01:55:54.932291 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.932257 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:55:54.932603 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.932558 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" podUID="0b6f67a0-2dae-400a-abd7-7c65fbdac225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 01:55:54.996113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.987827 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:55:54.996113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:54.987888 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" podUID="a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 01:55:55.060115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:55.059472 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 01:55:55.114082 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:55.114038 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" podUID="cb036aa6-48d5-465a-a4c8-5860ce3d69eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 01:55:55.114967 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:55.114934 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:55.159358 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:55.159254 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:55.179785 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:55.179728 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:55.232123 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:55.231149 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:55.232298 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:55.232218 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 01:55:55.246151 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:55.243597 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 01:55:55.356527 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:55.352437 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:55.452539 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:55.428365 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 01:55:55.671313 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:55.671283 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 01:55:55.783203 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:55.776845 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 01:55:55.873782 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:55:55.873728 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:55:55.929394 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:55.929358 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:55:55.987131 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:55.983627 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" podUID="0b6f67a0-2dae-400a-abd7-7c65fbdac225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 01:55:55.987131 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:55.969627 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:55:55.987131 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:55.983741 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" podUID="a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 01:55:56.039776 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:56.039687 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 01:55:56.049687 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:56.049654 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:55:56.273460 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:56.273427 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 01:55:56.396875 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:56.396771 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 01:55:56.497557 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:56.497517 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:56.497939 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:56.497912 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:56.912802 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:56.912768 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 01:55:56.917633 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:56.917601 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 01:55:56.917912 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:56.917878 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" podUID="0b6f67a0-2dae-400a-abd7-7c65fbdac225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 01:55:56.965509 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:56.964337 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 01:55:57.579848 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.579005 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:57.580335 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.580123 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:57.582275 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.581408 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 01:55:57.722881 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.722506 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:55:57.778002 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.777966 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-envoy\") pod \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " Oct 30 01:55:57.778204 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.778021 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-token\" (UniqueName: \"kubernetes.io/projected/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-token\") pod \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " Oct 30 01:55:57.778204 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.778127 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-workload-socket\") pod \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " Oct 30 01:55:57.778204 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.778160 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-workload-certs\") pod \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " Oct 30 01:55:57.778204 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.778193 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-kube-api-access-rhwsz\" (UniqueName: \"kubernetes.io/projected/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-kube-api-access-rhwsz\") pod \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " Oct 30 01:55:57.778497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.778240 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-data\") pod \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " Oct 30 01:55:57.778497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.778309 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-credential-socket\") pod \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " Oct 30 01:55:57.778497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.778347 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-podinfo\") pod \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " Oct 30 01:55:57.778497 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.778398 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istiod-ca-cert\") pod \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\" (UID: \"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b\") " Oct 30 01:55:57.780723 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.779615 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-workload-socket" pod "8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" (UID: "8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:57.780723 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.779772 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-data" pod "8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" (UID: "8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:57.780723 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.780667 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-credential-socket" pod "8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" (UID: "8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:57.781170 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.781138 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istiod-ca-cert" pod "8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" (UID: "8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:55:57.781421 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.781405 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-workload-certs" pod "8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" (UID: "8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:57.784271 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.784207 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-kube-api-access-rhwsz" pod "8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" (UID: "8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b"). InnerVolumeSpecName "kube-api-access-rhwsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:57.784403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.784304 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-podinfo" pod "8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" (UID: "8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:55:57.786342 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.786306 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-envoy" pod "8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" (UID: "8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:57.786628 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.786601 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-token" pod "8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" (UID: "8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:57.880884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.880848 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:55:57.880884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.880880 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:55:57.880884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.880889 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhwsz\" (UniqueName: \"kubernetes.io/projected/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-kube-api-access-rhwsz\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:55:57.881234 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.880899 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:55:57.881234 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.880908 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:55:57.881234 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.880916 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:55:57.881234 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.880924 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:55:57.881234 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.880934 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:55:57.881234 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.880941 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:55:57.915106 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.915054 5998 generic.go:356] "Generic (PLEG): container finished" podID="8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" containerID="b95b07aef71884a8226ca6fa15f5289872fef618cbbc949bf88e5dd6ffb6300d" exitCode=0 Oct 30 01:55:57.915284 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.915126 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" Oct 30 01:55:57.915284 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.915141 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" event={"ID":"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b","Type":"ContainerDied","Data":"b95b07aef71884a8226ca6fa15f5289872fef618cbbc949bf88e5dd6ffb6300d"} Oct 30 01:55:57.915284 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.915217 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8" event={"ID":"8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b","Type":"ContainerDied","Data":"6eda81a081146deb40ea184cd72bfc26a9fd4d22b124b7dd82399fd16d6e0051"} Oct 30 01:55:57.915284 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.915249 5998 scope.go:122] "RemoveContainer" containerID="b95b07aef71884a8226ca6fa15f5289872fef618cbbc949bf88e5dd6ffb6300d" Oct 30 01:55:57.943380 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.943349 5998 scope.go:122] "RemoveContainer" containerID="b95b07aef71884a8226ca6fa15f5289872fef618cbbc949bf88e5dd6ffb6300d" Oct 30 01:55:57.943783 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:55:57.943753 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95b07aef71884a8226ca6fa15f5289872fef618cbbc949bf88e5dd6ffb6300d\": container with ID starting with b95b07aef71884a8226ca6fa15f5289872fef618cbbc949bf88e5dd6ffb6300d not found: ID does not exist" containerID="b95b07aef71884a8226ca6fa15f5289872fef618cbbc949bf88e5dd6ffb6300d" Oct 30 01:55:57.943900 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.943791 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95b07aef71884a8226ca6fa15f5289872fef618cbbc949bf88e5dd6ffb6300d"} err="failed to get container status \"b95b07aef71884a8226ca6fa15f5289872fef618cbbc949bf88e5dd6ffb6300d\": rpc error: code = NotFound desc = could not find container \"b95b07aef71884a8226ca6fa15f5289872fef618cbbc949bf88e5dd6ffb6300d\": container with ID starting with b95b07aef71884a8226ca6fa15f5289872fef618cbbc949bf88e5dd6ffb6300d not found: ID does not exist" Oct 30 01:55:57.972515 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.972479 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8"] Oct 30 01:55:57.979197 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:57.979159 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2198e4-openshift-default-5fdf99b4c-btkz8"] Oct 30 01:55:58.015374 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:58.015271 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:55:58.552115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:58.551493 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" path="/var/lib/kubelet/pods/8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b/volumes" Oct 30 01:55:58.958114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:58.957328 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:55:58.958114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:58.957416 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" podUID="0b6f67a0-2dae-400a-abd7-7c65fbdac225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:55:59.207779 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:59.207744 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:59.208353 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:59.208283 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:59.210273 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:59.210248 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 01:55:59.932673 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:55:59.932636 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:00.171830 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.171787 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5"] Oct 30 01:56:00.172632 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.172609 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" containerName="istio-proxy" Oct 30 01:56:00.172739 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.172633 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" containerName="istio-proxy" Oct 30 01:56:00.172815 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.172800 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2f69ea-77d6-4b18-bb09-a6ca90e7d51b" containerName="istio-proxy" Oct 30 01:56:00.177721 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.177700 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5" Oct 30 01:56:00.203580 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.203553 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:56:00.203580 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.203567 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:56:00.203758 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.203644 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:56:00.203758 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.203684 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:56:00.208929 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.208904 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5"] Oct 30 01:56:00.213179 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.213084 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7ac19d-73e6-4435-b070-16d6db455913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-970f9605-rhhg5\" (UID: \"eb7ac19d-73e6-4435-b070-16d6db455913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5" Oct 30 01:56:00.213371 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.213352 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59vzl\" (UniqueName: \"kubernetes.io/projected/eb7ac19d-73e6-4435-b070-16d6db455913-kube-api-access-59vzl\") pod \"maas-api-revoke-sub-970f9605-rhhg5\" (UID: \"eb7ac19d-73e6-4435-b070-16d6db455913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5" Oct 30 01:56:00.213505 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.213488 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7ac19d-73e6-4435-b070-16d6db455913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-970f9605-rhhg5\" (UID: \"eb7ac19d-73e6-4435-b070-16d6db455913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5" Oct 30 01:56:00.314643 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.314605 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59vzl\" (UniqueName: \"kubernetes.io/projected/eb7ac19d-73e6-4435-b070-16d6db455913-kube-api-access-59vzl\") pod \"maas-api-revoke-sub-970f9605-rhhg5\" (UID: \"eb7ac19d-73e6-4435-b070-16d6db455913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5" Oct 30 01:56:00.314863 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.314710 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7ac19d-73e6-4435-b070-16d6db455913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-970f9605-rhhg5\" (UID: \"eb7ac19d-73e6-4435-b070-16d6db455913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5" Oct 30 01:56:00.314863 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.314780 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7ac19d-73e6-4435-b070-16d6db455913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-970f9605-rhhg5\" (UID: \"eb7ac19d-73e6-4435-b070-16d6db455913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5" Oct 30 01:56:00.315668 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.315642 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7ac19d-73e6-4435-b070-16d6db455913-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-970f9605-rhhg5\" (UID: \"eb7ac19d-73e6-4435-b070-16d6db455913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5" Oct 30 01:56:00.317228 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.317205 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7ac19d-73e6-4435-b070-16d6db455913-maas-api-xks-ca\") pod \"maas-api-revoke-sub-970f9605-rhhg5\" (UID: \"eb7ac19d-73e6-4435-b070-16d6db455913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5" Oct 30 01:56:00.337743 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.337712 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59vzl\" (UniqueName: \"kubernetes.io/projected/eb7ac19d-73e6-4435-b070-16d6db455913-kube-api-access-59vzl\") pod \"maas-api-revoke-sub-970f9605-rhhg5\" (UID: \"eb7ac19d-73e6-4435-b070-16d6db455913\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5" Oct 30 01:56:00.417627 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.417591 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:00.491871 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.491789 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5" Oct 30 01:56:00.563198 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.563156 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:00.862897 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.862867 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5"] Oct 30 01:56:00.864957 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:56:00.864928 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7ac19d_73e6_4435_b070_16d6db455913.slice/crio-e04bc61774a754fc055873186f4443dd6eaeee5856143fc79e25a34a2688003b WatchSource:0}: Error finding container e04bc61774a754fc055873186f4443dd6eaeee5856143fc79e25a34a2688003b: Status 404 returned error can't find the container with id e04bc61774a754fc055873186f4443dd6eaeee5856143fc79e25a34a2688003b Oct 30 01:56:00.889161 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.889133 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:00.954166 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:00.954117 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5" event={"ID":"eb7ac19d-73e6-4435-b070-16d6db455913","Type":"ContainerStarted","Data":"e04bc61774a754fc055873186f4443dd6eaeee5856143fc79e25a34a2688003b"} Oct 30 01:56:01.895598 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:01.895563 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:01.968158 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:01.968082 5998 generic.go:356] "Generic (PLEG): container finished" podID="eb7ac19d-73e6-4435-b070-16d6db455913" containerID="4d82dee480d17b4dde5967551ab2f2317662d073d8c1153e9e9d85153476eae6" exitCode=0 Oct 30 01:56:01.968322 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:01.968131 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5" event={"ID":"eb7ac19d-73e6-4435-b070-16d6db455913","Type":"ContainerDied","Data":"4d82dee480d17b4dde5967551ab2f2317662d073d8c1153e9e9d85153476eae6"} Oct 30 01:56:03.046807 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:03.046773 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk"] Oct 30 01:56:03.063489 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:03.063449 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d22ba8d6-cswkk"] Oct 30 01:56:03.170271 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:03.170243 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5" Oct 30 01:56:03.274143 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:03.274021 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb7ac19d-73e6-4435-b070-16d6db455913-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7ac19d-73e6-4435-b070-16d6db455913-maas-api-xks-ca\") pod \"eb7ac19d-73e6-4435-b070-16d6db455913\" (UID: \"eb7ac19d-73e6-4435-b070-16d6db455913\") " Oct 30 01:56:03.274143 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:03.274119 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb7ac19d-73e6-4435-b070-16d6db455913-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7ac19d-73e6-4435-b070-16d6db455913-maas-api-openshift-ca\") pod \"eb7ac19d-73e6-4435-b070-16d6db455913\" (UID: \"eb7ac19d-73e6-4435-b070-16d6db455913\") " Oct 30 01:56:03.274339 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:03.274227 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb7ac19d-73e6-4435-b070-16d6db455913-kube-api-access-59vzl\" (UniqueName: \"kubernetes.io/projected/eb7ac19d-73e6-4435-b070-16d6db455913-kube-api-access-59vzl\") pod \"eb7ac19d-73e6-4435-b070-16d6db455913\" (UID: \"eb7ac19d-73e6-4435-b070-16d6db455913\") " Oct 30 01:56:03.274495 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:03.274472 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb7ac19d-73e6-4435-b070-16d6db455913-maas-api-openshift-ca" pod "eb7ac19d-73e6-4435-b070-16d6db455913" (UID: "eb7ac19d-73e6-4435-b070-16d6db455913"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:56:03.274885 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:03.274863 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7ac19d-73e6-4435-b070-16d6db455913-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:56:03.276306 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:03.276281 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb7ac19d-73e6-4435-b070-16d6db455913-maas-api-xks-ca" pod "eb7ac19d-73e6-4435-b070-16d6db455913" (UID: "eb7ac19d-73e6-4435-b070-16d6db455913"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:56:03.276306 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:03.276282 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7ac19d-73e6-4435-b070-16d6db455913-kube-api-access-59vzl" pod "eb7ac19d-73e6-4435-b070-16d6db455913" (UID: "eb7ac19d-73e6-4435-b070-16d6db455913"). InnerVolumeSpecName "kube-api-access-59vzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:56:03.376003 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:03.375962 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7ac19d-73e6-4435-b070-16d6db455913-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:56:03.376003 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:03.375999 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59vzl\" (UniqueName: \"kubernetes.io/projected/eb7ac19d-73e6-4435-b070-16d6db455913-kube-api-access-59vzl\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:56:03.686110 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:03.686063 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:03.984464 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:03.984383 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5" Oct 30 01:56:03.984464 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:03.984418 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5" event={"ID":"eb7ac19d-73e6-4435-b070-16d6db455913","Type":"ContainerDied","Data":"e04bc61774a754fc055873186f4443dd6eaeee5856143fc79e25a34a2688003b"} Oct 30 01:56:03.984654 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:03.984471 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e04bc61774a754fc055873186f4443dd6eaeee5856143fc79e25a34a2688003b" Oct 30 01:56:04.540549 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:04.540508 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d520569f-2b5b-4bb4-862e-6042401aadf8" path="/var/lib/kubelet/pods/d520569f-2b5b-4bb4-862e-6042401aadf8/volumes" Oct 30 01:56:05.378169 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:05.378133 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:07.080305 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:07.080264 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:08.495251 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:08.495209 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:09.138029 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:09.137990 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:10.765749 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:10.765715 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:10.921113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:10.918663 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:11.448114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:11.446958 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:11.879595 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:11.879508 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:12.937996 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:12.937936 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:14.823072 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:14.823024 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:15.129800 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:15.129716 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:15.266313 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:15.266270 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:16.265795 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.265760 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n"] Oct 30 01:56:16.266399 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.266383 5998 cpu_manager.go:414] "containerMap: removing container" podUID="eb7ac19d-73e6-4435-b070-16d6db455913" containerName="revoke-keys" Oct 30 01:56:16.266455 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.266400 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb7ac19d-73e6-4435-b070-16d6db455913" containerName="revoke-keys" Oct 30 01:56:16.266540 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.266530 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb7ac19d-73e6-4435-b070-16d6db455913" containerName="revoke-keys" Oct 30 01:56:16.271640 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.271621 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n" Oct 30 01:56:16.293039 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.293001 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n"] Oct 30 01:56:16.297609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.297528 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:56:16.297609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.297549 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:56:16.297609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.297563 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:56:16.297886 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.297715 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:56:16.333958 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.333918 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c95a7cd6-6610-4f48-85cd-2011f3a2106a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n\" (UID: \"c95a7cd6-6610-4f48-85cd-2011f3a2106a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n" Oct 30 01:56:16.334264 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.334035 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkf6k\" (UniqueName: \"kubernetes.io/projected/c95a7cd6-6610-4f48-85cd-2011f3a2106a-kube-api-access-tkf6k\") pod \"maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n\" (UID: \"c95a7cd6-6610-4f48-85cd-2011f3a2106a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n" Oct 30 01:56:16.334264 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.334250 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c95a7cd6-6610-4f48-85cd-2011f3a2106a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n\" (UID: \"c95a7cd6-6610-4f48-85cd-2011f3a2106a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n" Oct 30 01:56:16.429178 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.429142 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:16.435485 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.435453 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkf6k\" (UniqueName: \"kubernetes.io/projected/c95a7cd6-6610-4f48-85cd-2011f3a2106a-kube-api-access-tkf6k\") pod \"maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n\" (UID: \"c95a7cd6-6610-4f48-85cd-2011f3a2106a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n" Oct 30 01:56:16.435665 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.435627 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c95a7cd6-6610-4f48-85cd-2011f3a2106a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n\" (UID: \"c95a7cd6-6610-4f48-85cd-2011f3a2106a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n" Oct 30 01:56:16.435784 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.435766 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c95a7cd6-6610-4f48-85cd-2011f3a2106a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n\" (UID: \"c95a7cd6-6610-4f48-85cd-2011f3a2106a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n" Oct 30 01:56:16.437198 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.437028 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c95a7cd6-6610-4f48-85cd-2011f3a2106a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n\" (UID: \"c95a7cd6-6610-4f48-85cd-2011f3a2106a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n" Oct 30 01:56:16.453974 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.453923 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c95a7cd6-6610-4f48-85cd-2011f3a2106a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n\" (UID: \"c95a7cd6-6610-4f48-85cd-2011f3a2106a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n" Oct 30 01:56:16.460047 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.460021 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkf6k\" (UniqueName: \"kubernetes.io/projected/c95a7cd6-6610-4f48-85cd-2011f3a2106a-kube-api-access-tkf6k\") pod \"maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n\" (UID: \"c95a7cd6-6610-4f48-85cd-2011f3a2106a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n" Oct 30 01:56:16.586762 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.586668 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n" Oct 30 01:56:16.603784 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:16.603705 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:17.033528 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:17.033469 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n"] Oct 30 01:56:17.181669 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:17.181633 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n" event={"ID":"c95a7cd6-6610-4f48-85cd-2011f3a2106a","Type":"ContainerStarted","Data":"1aa377d4c7cad2f3454fd769429a5be9d53f090cc2957d29845de97820872d58"} Oct 30 01:56:17.297942 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:17.297864 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:17.654162 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:17.653339 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:17.889949 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:17.889855 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:18.197604 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:18.197564 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:18.274194 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:18.274043 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n" event={"ID":"c95a7cd6-6610-4f48-85cd-2011f3a2106a","Type":"ContainerStarted","Data":"e66d0c11813d8941b2eda296521bd3e2f20f756061abd3b9e638f82c95846791"} Oct 30 01:56:18.351616 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:18.351252 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n" podStartSLOduration=2.351230187 podStartE2EDuration="2.351230187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:18.34642982 +0000 UTC m=+1607.925220904" watchObservedRunningTime="2026-10-30 01:56:18.351230187 +0000 UTC m=+1607.930021267" Oct 30 01:56:19.282858 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:19.282822 5998 generic.go:356] "Generic (PLEG): container finished" podID="c95a7cd6-6610-4f48-85cd-2011f3a2106a" containerID="e66d0c11813d8941b2eda296521bd3e2f20f756061abd3b9e638f82c95846791" exitCode=0 Oct 30 01:56:19.283152 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:19.282909 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n" event={"ID":"c95a7cd6-6610-4f48-85cd-2011f3a2106a","Type":"ContainerDied","Data":"e66d0c11813d8941b2eda296521bd3e2f20f756061abd3b9e638f82c95846791"} Oct 30 01:56:19.284776 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:19.284751 5998 generic.go:356] "Generic (PLEG): container finished" podID="9d11c0bb-7924-485b-8514-1213bc06afca" containerID="1ff026f8f2c1777effacccd9fef323c9c2ccd7c3b93bf51c04a605ff21594ed9" exitCode=0 Oct 30 01:56:19.284885 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:19.284824 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" event={"ID":"9d11c0bb-7924-485b-8514-1213bc06afca","Type":"ContainerDied","Data":"1ff026f8f2c1777effacccd9fef323c9c2ccd7c3b93bf51c04a605ff21594ed9"} Oct 30 01:56:19.285276 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:19.285262 5998 scope.go:122] "RemoveContainer" containerID="1ff026f8f2c1777effacccd9fef323c9c2ccd7c3b93bf51c04a605ff21594ed9" Oct 30 01:56:19.286919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:19.286892 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:56:19.286988 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:19.286975 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:56:19.287029 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:19.287009 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:56:19.557721 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:19.557625 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:56:20.171109 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:20.171056 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:20.289241 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:20.289211 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:20.297961 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:20.297926 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" event={"ID":"9d11c0bb-7924-485b-8514-1213bc06afca","Type":"ContainerStarted","Data":"9578979d4d878341adda97a7468e9354e33345c9d419bbaee3de14f5624c3850"} Oct 30 01:56:20.482744 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:20.482633 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:20.579945 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:20.579886 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n" Oct 30 01:56:20.581937 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:20.581910 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c95a7cd6-6610-4f48-85cd-2011f3a2106a-kube-api-access-tkf6k\" (UniqueName: \"kubernetes.io/projected/c95a7cd6-6610-4f48-85cd-2011f3a2106a-kube-api-access-tkf6k\") pod \"c95a7cd6-6610-4f48-85cd-2011f3a2106a\" (UID: \"c95a7cd6-6610-4f48-85cd-2011f3a2106a\") " Oct 30 01:56:20.582040 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:20.581944 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c95a7cd6-6610-4f48-85cd-2011f3a2106a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c95a7cd6-6610-4f48-85cd-2011f3a2106a-maas-api-openshift-ca\") pod \"c95a7cd6-6610-4f48-85cd-2011f3a2106a\" (UID: \"c95a7cd6-6610-4f48-85cd-2011f3a2106a\") " Oct 30 01:56:20.582040 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:20.581965 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c95a7cd6-6610-4f48-85cd-2011f3a2106a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c95a7cd6-6610-4f48-85cd-2011f3a2106a-maas-api-xks-ca\") pod \"c95a7cd6-6610-4f48-85cd-2011f3a2106a\" (UID: \"c95a7cd6-6610-4f48-85cd-2011f3a2106a\") " Oct 30 01:56:20.582348 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:20.582326 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c95a7cd6-6610-4f48-85cd-2011f3a2106a-maas-api-openshift-ca" pod "c95a7cd6-6610-4f48-85cd-2011f3a2106a" (UID: "c95a7cd6-6610-4f48-85cd-2011f3a2106a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:56:20.582660 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:20.582642 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c95a7cd6-6610-4f48-85cd-2011f3a2106a-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:56:20.584159 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:20.584134 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c95a7cd6-6610-4f48-85cd-2011f3a2106a-kube-api-access-tkf6k" pod "c95a7cd6-6610-4f48-85cd-2011f3a2106a" (UID: "c95a7cd6-6610-4f48-85cd-2011f3a2106a"). InnerVolumeSpecName "kube-api-access-tkf6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:56:20.584247 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:20.584187 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c95a7cd6-6610-4f48-85cd-2011f3a2106a-maas-api-xks-ca" pod "c95a7cd6-6610-4f48-85cd-2011f3a2106a" (UID: "c95a7cd6-6610-4f48-85cd-2011f3a2106a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:56:20.683570 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:20.683530 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkf6k\" (UniqueName: \"kubernetes.io/projected/c95a7cd6-6610-4f48-85cd-2011f3a2106a-kube-api-access-tkf6k\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:56:20.683570 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:20.683564 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c95a7cd6-6610-4f48-85cd-2011f3a2106a-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:56:21.117116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:21.116367 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:56:21.117116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:21.116498 5998 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:56:21.117116 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:21.116543 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" podUID="9d11c0bb-7924-485b-8514-1213bc06afca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 01:56:21.333133 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:21.333072 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n" Oct 30 01:56:21.333350 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:21.333141 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n" event={"ID":"c95a7cd6-6610-4f48-85cd-2011f3a2106a","Type":"ContainerDied","Data":"1aa377d4c7cad2f3454fd769429a5be9d53f090cc2957d29845de97820872d58"} Oct 30 01:56:21.333350 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:21.333186 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aa377d4c7cad2f3454fd769429a5be9d53f090cc2957d29845de97820872d58" Oct 30 01:56:22.036938 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:22.036904 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b"] Oct 30 01:56:22.048992 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:22.048963 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1683ded2-6v98b"] Oct 30 01:56:22.122114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:22.119454 5998 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:56:22.122114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:22.119537 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" podUID="9d11c0bb-7924-485b-8514-1213bc06afca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 01:56:22.384811 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:22.384726 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:22.505990 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:22.504488 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:22.541564 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:22.541526 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d78788f8-0311-47f6-9126-afa411a940f2" path="/var/lib/kubelet/pods/d78788f8-0311-47f6-9126-afa411a940f2/volumes" Oct 30 01:56:22.662537 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:22.661605 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:23.125169 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:23.124491 5998 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:56:23.125169 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:23.124556 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" podUID="9d11c0bb-7924-485b-8514-1213bc06afca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 01:56:24.120115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:24.118478 5998 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:56:24.120115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:24.118545 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" podUID="9d11c0bb-7924-485b-8514-1213bc06afca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 01:56:24.363179 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:24.362912 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:24.476466 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:24.476430 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:25.129961 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:25.124627 5998 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:56:25.129961 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:25.124689 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" podUID="9d11c0bb-7924-485b-8514-1213bc06afca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 01:56:25.247113 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:56:25.246558 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:56:25.365641 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:25.365607 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:26.111174 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:26.111142 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:26.119890 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:26.119865 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:56:26.120376 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:26.120354 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:56:26.121147 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:26.121128 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:56:26.219486 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:26.218238 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:26.813079 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:26.813049 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:26.925163 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:26.925127 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:27.040633 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:27.036457 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:27.184837 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:27.184807 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:27.292237 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:27.292186 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:29.190755 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:29.190718 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:30.778571 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:30.778533 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:31.519519 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:31.519483 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:56:32.034797 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:32.034762 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj"] Oct 30 01:56:32.046252 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:32.046224 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a90a24a-hzxjj"] Oct 30 01:56:32.565592 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:32.565551 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d9393ac-db00-44c6-9fd3-8f767c77a796" path="/var/lib/kubelet/pods/2d9393ac-db00-44c6-9fd3-8f767c77a796/volumes" Oct 30 01:56:32.572390 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:32.572357 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:32.800508 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:32.800468 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:34.802063 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:34.802011 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:34.922835 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:34.922797 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:35.057453 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:35.057343 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" podUID="cb036aa6-48d5-465a-a4c8-5860ce3d69eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:56:35.085227 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:35.085182 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:56:35.175245 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:35.175195 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-zghmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:56:35.175558 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:35.175525 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" podUID="94053f0f-2fd8-4ab0-914a-256ddadb5d85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:56:35.189280 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:35.189235 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-57s4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:56:35.189612 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:35.189576 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" podUID="caaec13d-e62d-4450-864c-d9e3fb7ca8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:56:35.190783 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:35.190756 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:56:35.216605 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:35.216561 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:56:35.247231 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:35.247186 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:56:35.891114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:35.890996 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:56:36.421227 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.421191 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc"] Oct 30 01:56:36.421870 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.421853 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c95a7cd6-6610-4f48-85cd-2011f3a2106a" containerName="revoke-keys" Oct 30 01:56:36.421917 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.421872 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c95a7cd6-6610-4f48-85cd-2011f3a2106a" containerName="revoke-keys" Oct 30 01:56:36.421966 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.421956 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c95a7cd6-6610-4f48-85cd-2011f3a2106a" containerName="revoke-keys" Oct 30 01:56:36.427078 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.427062 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc" Oct 30 01:56:36.449635 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.449604 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc"] Oct 30 01:56:36.454682 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.454546 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:56:36.454682 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.454581 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:56:36.454682 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.454627 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:56:36.454682 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.454654 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:56:36.509357 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.509325 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79834ac-25b6-4c70-b68f-2ee4228f6001-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e82ba923-zwlpc\" (UID: \"c79834ac-25b6-4c70-b68f-2ee4228f6001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc" Oct 30 01:56:36.509526 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.509374 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qxg8\" (UniqueName: \"kubernetes.io/projected/c79834ac-25b6-4c70-b68f-2ee4228f6001-kube-api-access-8qxg8\") pod \"maas-api-revoke-sub-e82ba923-zwlpc\" (UID: \"c79834ac-25b6-4c70-b68f-2ee4228f6001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc" Oct 30 01:56:36.509526 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.509398 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79834ac-25b6-4c70-b68f-2ee4228f6001-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e82ba923-zwlpc\" (UID: \"c79834ac-25b6-4c70-b68f-2ee4228f6001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc" Oct 30 01:56:36.610619 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.610588 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79834ac-25b6-4c70-b68f-2ee4228f6001-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e82ba923-zwlpc\" (UID: \"c79834ac-25b6-4c70-b68f-2ee4228f6001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc" Oct 30 01:56:36.610799 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.610637 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qxg8\" (UniqueName: \"kubernetes.io/projected/c79834ac-25b6-4c70-b68f-2ee4228f6001-kube-api-access-8qxg8\") pod \"maas-api-revoke-sub-e82ba923-zwlpc\" (UID: \"c79834ac-25b6-4c70-b68f-2ee4228f6001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc" Oct 30 01:56:36.610799 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.610676 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79834ac-25b6-4c70-b68f-2ee4228f6001-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e82ba923-zwlpc\" (UID: \"c79834ac-25b6-4c70-b68f-2ee4228f6001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc" Oct 30 01:56:36.611401 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.611370 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79834ac-25b6-4c70-b68f-2ee4228f6001-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e82ba923-zwlpc\" (UID: \"c79834ac-25b6-4c70-b68f-2ee4228f6001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc" Oct 30 01:56:36.613009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.612980 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79834ac-25b6-4c70-b68f-2ee4228f6001-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e82ba923-zwlpc\" (UID: \"c79834ac-25b6-4c70-b68f-2ee4228f6001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc" Oct 30 01:56:36.634645 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.634604 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qxg8\" (UniqueName: \"kubernetes.io/projected/c79834ac-25b6-4c70-b68f-2ee4228f6001-kube-api-access-8qxg8\") pod \"maas-api-revoke-sub-e82ba923-zwlpc\" (UID: \"c79834ac-25b6-4c70-b68f-2ee4228f6001\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc" Oct 30 01:56:36.683302 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.683219 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:36.742463 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:36.742418 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc" Oct 30 01:56:37.063957 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:37.063926 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc"] Oct 30 01:56:37.066408 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:56:37.066371 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79834ac_25b6_4c70_b68f_2ee4228f6001.slice/crio-1685eabb3d41b296cfe190e859c4ed0ce4cee0177c4e7ad504e53aecf67f24ca WatchSource:0}: Error finding container 1685eabb3d41b296cfe190e859c4ed0ce4cee0177c4e7ad504e53aecf67f24ca: Status 404 returned error can't find the container with id 1685eabb3d41b296cfe190e859c4ed0ce4cee0177c4e7ad504e53aecf67f24ca Oct 30 01:56:37.669158 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:37.669084 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc" event={"ID":"c79834ac-25b6-4c70-b68f-2ee4228f6001","Type":"ContainerStarted","Data":"90120c19222e8cc32b23ff74e79305442dfe53f930aba61cffa17e905c38b44e"} Oct 30 01:56:37.669342 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:37.669180 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc" event={"ID":"c79834ac-25b6-4c70-b68f-2ee4228f6001","Type":"ContainerStarted","Data":"1685eabb3d41b296cfe190e859c4ed0ce4cee0177c4e7ad504e53aecf67f24ca"} Oct 30 01:56:37.718600 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:37.718540 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc" podStartSLOduration=1.718522379 podStartE2EDuration="1.718522379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:37.714724828 +0000 UTC m=+1627.293515911" watchObservedRunningTime="2026-10-30 01:56:37.718522379 +0000 UTC m=+1627.297313459" Oct 30 01:56:38.332057 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:38.331950 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:38.456225 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:38.452397 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:38.678016 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:38.677980 5998 generic.go:356] "Generic (PLEG): container finished" podID="c79834ac-25b6-4c70-b68f-2ee4228f6001" containerID="90120c19222e8cc32b23ff74e79305442dfe53f930aba61cffa17e905c38b44e" exitCode=0 Oct 30 01:56:38.678248 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:38.678132 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc" event={"ID":"c79834ac-25b6-4c70-b68f-2ee4228f6001","Type":"ContainerDied","Data":"90120c19222e8cc32b23ff74e79305442dfe53f930aba61cffa17e905c38b44e"} Oct 30 01:56:38.792006 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:38.791972 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:39.852189 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:39.852163 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc" Oct 30 01:56:39.958268 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:39.958154 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c79834ac-25b6-4c70-b68f-2ee4228f6001-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79834ac-25b6-4c70-b68f-2ee4228f6001-maas-api-openshift-ca\") pod \"c79834ac-25b6-4c70-b68f-2ee4228f6001\" (UID: \"c79834ac-25b6-4c70-b68f-2ee4228f6001\") " Oct 30 01:56:39.958572 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:39.958383 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c79834ac-25b6-4c70-b68f-2ee4228f6001-kube-api-access-8qxg8\" (UniqueName: \"kubernetes.io/projected/c79834ac-25b6-4c70-b68f-2ee4228f6001-kube-api-access-8qxg8\") pod \"c79834ac-25b6-4c70-b68f-2ee4228f6001\" (UID: \"c79834ac-25b6-4c70-b68f-2ee4228f6001\") " Oct 30 01:56:39.958652 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:39.958597 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c79834ac-25b6-4c70-b68f-2ee4228f6001-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79834ac-25b6-4c70-b68f-2ee4228f6001-maas-api-xks-ca\") pod \"c79834ac-25b6-4c70-b68f-2ee4228f6001\" (UID: \"c79834ac-25b6-4c70-b68f-2ee4228f6001\") " Oct 30 01:56:39.959552 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:39.959521 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79834ac-25b6-4c70-b68f-2ee4228f6001-maas-api-openshift-ca" pod "c79834ac-25b6-4c70-b68f-2ee4228f6001" (UID: "c79834ac-25b6-4c70-b68f-2ee4228f6001"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:56:39.961324 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:39.961297 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79834ac-25b6-4c70-b68f-2ee4228f6001-kube-api-access-8qxg8" pod "c79834ac-25b6-4c70-b68f-2ee4228f6001" (UID: "c79834ac-25b6-4c70-b68f-2ee4228f6001"). InnerVolumeSpecName "kube-api-access-8qxg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:56:39.961487 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:39.961378 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c79834ac-25b6-4c70-b68f-2ee4228f6001-maas-api-xks-ca" pod "c79834ac-25b6-4c70-b68f-2ee4228f6001" (UID: "c79834ac-25b6-4c70-b68f-2ee4228f6001"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:56:40.060854 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:40.060816 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c79834ac-25b6-4c70-b68f-2ee4228f6001-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:56:40.060854 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:40.060852 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c79834ac-25b6-4c70-b68f-2ee4228f6001-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:56:40.061142 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:40.060868 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qxg8\" (UniqueName: \"kubernetes.io/projected/c79834ac-25b6-4c70-b68f-2ee4228f6001-kube-api-access-8qxg8\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:56:40.278879 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:40.278791 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:40.703254 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:40.703222 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc" Oct 30 01:56:40.703455 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:40.703246 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc" event={"ID":"c79834ac-25b6-4c70-b68f-2ee4228f6001","Type":"ContainerDied","Data":"1685eabb3d41b296cfe190e859c4ed0ce4cee0177c4e7ad504e53aecf67f24ca"} Oct 30 01:56:40.703455 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:40.703333 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1685eabb3d41b296cfe190e859c4ed0ce4cee0177c4e7ad504e53aecf67f24ca" Oct 30 01:56:41.892108 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:41.892013 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:43.483934 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:43.483897 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:45.090874 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:45.090834 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:47.292788 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:47.292751 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:49.026894 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:49.026850 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:50.154849 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.154807 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd"] Oct 30 01:56:50.155466 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.155450 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c79834ac-25b6-4c70-b68f-2ee4228f6001" containerName="revoke-keys" Oct 30 01:56:50.155515 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.155467 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c79834ac-25b6-4c70-b68f-2ee4228f6001" containerName="revoke-keys" Oct 30 01:56:50.155602 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.155593 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c79834ac-25b6-4c70-b68f-2ee4228f6001" containerName="revoke-keys" Oct 30 01:56:50.159542 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.159521 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd" Oct 30 01:56:50.181480 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.181444 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd"] Oct 30 01:56:50.187769 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.187746 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:56:50.187914 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.187763 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:56:50.187914 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.187816 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 01:56:50.187914 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.187829 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:56:50.212320 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.212288 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp26m\" (UniqueName: \"kubernetes.io/projected/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-kube-api-access-dp26m\") pod \"maas-api-revoke-sub-82b769c4-5v2rd\" (UID: \"e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd" Oct 30 01:56:50.212575 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.212543 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82b769c4-5v2rd\" (UID: \"e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd" Oct 30 01:56:50.212732 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.212606 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82b769c4-5v2rd\" (UID: \"e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd" Oct 30 01:56:50.313562 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.313520 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp26m\" (UniqueName: \"kubernetes.io/projected/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-kube-api-access-dp26m\") pod \"maas-api-revoke-sub-82b769c4-5v2rd\" (UID: \"e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd" Oct 30 01:56:50.313766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.313655 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82b769c4-5v2rd\" (UID: \"e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd" Oct 30 01:56:50.313766 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.313678 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82b769c4-5v2rd\" (UID: \"e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd" Oct 30 01:56:50.314359 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.314336 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82b769c4-5v2rd\" (UID: \"e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd" Oct 30 01:56:50.316050 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.316026 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82b769c4-5v2rd\" (UID: \"e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd" Oct 30 01:56:50.336788 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.336757 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp26m\" (UniqueName: \"kubernetes.io/projected/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-kube-api-access-dp26m\") pod \"maas-api-revoke-sub-82b769c4-5v2rd\" (UID: \"e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd" Oct 30 01:56:50.473831 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.473791 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd" Oct 30 01:56:50.934663 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:50.934596 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd"] Oct 30 01:56:50.938260 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:56:50.938225 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c5a893_d76d_417e_b3ec_76d6e5fc5d4f.slice/crio-351a631e2356e4b0a6a2025eeeee9963e1ed2993542fbc0e5bbd2d573a80f722 WatchSource:0}: Error finding container 351a631e2356e4b0a6a2025eeeee9963e1ed2993542fbc0e5bbd2d573a80f722: Status 404 returned error can't find the container with id 351a631e2356e4b0a6a2025eeeee9963e1ed2993542fbc0e5bbd2d573a80f722 Oct 30 01:56:51.061085 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:51.061050 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:51.799832 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:51.799795 5998 generic.go:356] "Generic (PLEG): container finished" podID="e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f" containerID="c9e02ad349dfad795f7407adf6adad3edaf5c94b744189b87a0c919142e07c36" exitCode=0 Oct 30 01:56:51.800269 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:51.799876 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd" event={"ID":"e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f","Type":"ContainerDied","Data":"c9e02ad349dfad795f7407adf6adad3edaf5c94b744189b87a0c919142e07c36"} Oct 30 01:56:51.800269 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:51.799930 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd" event={"ID":"e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f","Type":"ContainerStarted","Data":"351a631e2356e4b0a6a2025eeeee9963e1ed2993542fbc0e5bbd2d573a80f722"} Oct 30 01:56:52.979936 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:52.979909 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd" Oct 30 01:56:53.074077 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:53.073985 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-maas-api-xks-ca\") pod \"e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f\" (UID: \"e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f\") " Oct 30 01:56:53.074077 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:53.074038 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-maas-api-openshift-ca\") pod \"e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f\" (UID: \"e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f\") " Oct 30 01:56:53.074336 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:53.074106 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-kube-api-access-dp26m\" (UniqueName: \"kubernetes.io/projected/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-kube-api-access-dp26m\") pod \"e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f\" (UID: \"e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f\") " Oct 30 01:56:53.074483 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:53.074437 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-maas-api-openshift-ca" pod "e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f" (UID: "e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:56:53.074781 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:53.074759 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:56:53.076465 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:53.076437 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-kube-api-access-dp26m" pod "e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f" (UID: "e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f"). InnerVolumeSpecName "kube-api-access-dp26m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:56:53.076605 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:53.076589 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-maas-api-xks-ca" pod "e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f" (UID: "e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:56:53.081471 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:53.081445 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:53.179262 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:53.176578 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:56:53.179262 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:53.176613 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp26m\" (UniqueName: \"kubernetes.io/projected/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f-kube-api-access-dp26m\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:56:53.818047 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:53.818016 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd" Oct 30 01:56:53.818242 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:53.818061 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd" event={"ID":"e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f","Type":"ContainerDied","Data":"351a631e2356e4b0a6a2025eeeee9963e1ed2993542fbc0e5bbd2d573a80f722"} Oct 30 01:56:53.818242 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:53.818130 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="351a631e2356e4b0a6a2025eeeee9963e1ed2993542fbc0e5bbd2d573a80f722" Oct 30 01:56:54.789334 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:54.789304 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:55.221491 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:56:55.221446 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:56:55.671414 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:55.671364 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:55.787689 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:55.787653 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:55.898757 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:55.898721 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:56.310339 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:56.310302 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:56.517786 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:56.517754 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:57.227366 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.227317 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx"] Oct 30 01:56:57.228240 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.228214 5998 cpu_manager.go:414] "containerMap: removing container" podUID="e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f" containerName="revoke-keys" Oct 30 01:56:57.228424 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.228404 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f" containerName="revoke-keys" Oct 30 01:56:57.228596 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.228582 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f" containerName="revoke-keys" Oct 30 01:56:57.247988 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.239070 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.265042 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.265008 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx"] Oct 30 01:56:57.275436 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.275406 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2818a6-openshift-default-dockercfg-wjrv6\"" Oct 30 01:56:57.275815 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.275789 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:57.356630 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.353083 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-envoy\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.356630 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.353185 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-workload-socket\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.356630 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.353217 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-token\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.356630 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.353255 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istiod-ca-cert\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.356630 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.353299 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-workload-certs\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.356630 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.353334 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-data\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.356630 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.353375 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs7tl\" (UniqueName: \"kubernetes.io/projected/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-kube-api-access-cs7tl\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.356630 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.353463 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-credential-socket\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.356630 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.353504 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-podinfo\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.455018 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.454983 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-data\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.455242 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.455032 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs7tl\" (UniqueName: \"kubernetes.io/projected/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-kube-api-access-cs7tl\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.455242 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.455085 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-credential-socket\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.455242 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.455142 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-podinfo\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.455435 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.455277 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-envoy\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.455435 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.455310 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-workload-socket\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.455435 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.455331 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-token\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.455435 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.455355 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istiod-ca-cert\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.455435 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.455387 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-workload-certs\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.455684 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.455529 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-data\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.455684 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.455563 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-credential-socket\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.455684 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.455679 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-workload-certs\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.455871 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.455767 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-workload-socket\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.456254 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.456226 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istiod-ca-cert\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.457911 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.457883 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-envoy\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.458023 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.458006 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-podinfo\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.478274 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.478178 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-token\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.478401 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.478359 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs7tl\" (UniqueName: \"kubernetes.io/projected/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-kube-api-access-cs7tl\") pod \"e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.574915 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.574881 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:57.685954 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.685920 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:57.796274 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.796143 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:57.911975 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.911941 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx"] Oct 30 01:56:57.916241 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:56:57.916208 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa5cf05f_5444_459e_ba19_657e8c9f4ee1.slice/crio-2edfa97e651c60aed441496742ae317490cf0730144108be3b9b6f4a54357371 WatchSource:0}: Error finding container 2edfa97e651c60aed441496742ae317490cf0730144108be3b9b6f4a54357371: Status 404 returned error can't find the container with id 2edfa97e651c60aed441496742ae317490cf0730144108be3b9b6f4a54357371 Oct 30 01:56:57.919147 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.919106 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:56:57.919263 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.919177 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:56:57.919263 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:57.919210 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:56:58.319216 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:58.319180 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:58.910330 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:58.910288 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" event={"ID":"fa5cf05f-5444-459e-ba19-657e8c9f4ee1","Type":"ContainerStarted","Data":"e800b0c096f46733333b293bfa3c5d29fa9c7af0475ef21250822a59d77bd070"} Oct 30 01:56:58.910780 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:58.910755 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" event={"ID":"fa5cf05f-5444-459e-ba19-657e8c9f4ee1","Type":"ContainerStarted","Data":"2edfa97e651c60aed441496742ae317490cf0730144108be3b9b6f4a54357371"} Oct 30 01:56:58.979275 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:58.979205 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" podStartSLOduration=1.979186358 podStartE2EDuration="1.979186358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:58.966967711 +0000 UTC m=+1648.545758793" watchObservedRunningTime="2026-10-30 01:56:58.979186358 +0000 UTC m=+1648.557977440" Oct 30 01:56:59.226113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:59.224710 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:59.333657 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:59.333624 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:59.481862 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:59.481771 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:59.512807 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:59.512757 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n"] Oct 30 01:56:59.517276 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:59.517248 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2933ff00-vjk5n"] Oct 30 01:56:59.576024 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:59.575991 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:59.581768 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:59.581741 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:59.655366 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:59.655333 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:56:59.917616 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:59.917584 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:56:59.918573 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:56:59.918550 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:57:00.540203 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:00.540169 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c95a7cd6-6610-4f48-85cd-2011f3a2106a" path="/var/lib/kubelet/pods/c95a7cd6-6610-4f48-85cd-2011f3a2106a/volumes" Oct 30 01:57:01.410826 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:01.410789 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:01.527377 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:01.527340 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:01.643819 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:01.643770 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:02.103790 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.103752 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv"] Oct 30 01:57:02.109357 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.109327 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.140331 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.140297 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istiod-ca-cert\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.140493 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.140341 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-podinfo\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.140493 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.140369 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-token\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.140493 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.140413 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-workload-certs\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.140493 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.140434 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-credential-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.140719 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.140649 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7bwq\" (UniqueName: \"kubernetes.io/projected/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-kube-api-access-x7bwq\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.140780 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.140715 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-data\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.140780 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.140751 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-envoy\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.140880 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.140785 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-workload-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.153247 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.153202 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv"] Oct 30 01:57:02.242527 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.242486 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-workload-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.242731 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.242602 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istiod-ca-cert\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.242731 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.242635 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-podinfo\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.243233 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.243202 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-token\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.243357 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.243340 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-workload-certs\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.243435 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.243419 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-credential-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.243534 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.243508 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-workload-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.243837 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.243812 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-workload-certs\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.243982 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.243954 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7bwq\" (UniqueName: \"kubernetes.io/projected/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-kube-api-access-x7bwq\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.244174 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.244057 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-data\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.244174 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.244139 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-envoy\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.244361 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.244207 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-credential-socket\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.244361 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.244329 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istiod-ca-cert\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.244519 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.244498 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-data\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.245458 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.245439 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-podinfo\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.246358 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.246342 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-envoy\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.266381 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.266347 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-token\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.267003 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.266978 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7bwq\" (UniqueName: \"kubernetes.io/projected/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-kube-api-access-x7bwq\") pod \"e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.447504 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.447466 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:02.780769 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.780736 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv"] Oct 30 01:57:02.782338 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:57:02.782308 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf359d0e2_0a2f_4321_a1bd_c48bea8684b1.slice/crio-1014d4c0f7dfadd78f39cef612818fc69ee302a5db338f419c567fce5b2eb3ee WatchSource:0}: Error finding container 1014d4c0f7dfadd78f39cef612818fc69ee302a5db338f419c567fce5b2eb3ee: Status 404 returned error can't find the container with id 1014d4c0f7dfadd78f39cef612818fc69ee302a5db338f419c567fce5b2eb3ee Oct 30 01:57:02.785283 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.785247 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:57:02.785386 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.785336 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:57:02.785451 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.785385 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:57:02.949256 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.949209 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" event={"ID":"f359d0e2-0a2f-4321-a1bd-c48bea8684b1","Type":"ContainerStarted","Data":"a597da9e106ea78e072f9f7d784c7ed04f83ec63913441ba4f808778796c1ee1"} Oct 30 01:57:02.949453 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.949275 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" event={"ID":"f359d0e2-0a2f-4321-a1bd-c48bea8684b1","Type":"ContainerStarted","Data":"1014d4c0f7dfadd78f39cef612818fc69ee302a5db338f419c567fce5b2eb3ee"} Oct 30 01:57:02.998144 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:02.998069 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" podStartSLOduration=0.998055083 podStartE2EDuration="998.055083ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:57:02.993310156 +0000 UTC m=+1652.572101243" watchObservedRunningTime="2026-10-30 01:57:02.998055083 +0000 UTC m=+1652.576846163" Oct 30 01:57:03.298838 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:03.298803 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:03.451115 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:03.438672 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:03.457133 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:03.453462 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:03.457133 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:03.456359 5998 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 01:57:03.457133 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:03.456413 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" podUID="f359d0e2-0a2f-4321-a1bd-c48bea8684b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 01:57:03.569812 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:03.569722 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:03.860391 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:03.860295 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:04.448667 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:04.448631 5998 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 01:57:04.448851 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:04.448688 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" podUID="f359d0e2-0a2f-4321-a1bd-c48bea8684b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 01:57:05.285584 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:05.285541 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:05.417483 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:05.411726 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:05.453063 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:05.453023 5998 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 01:57:05.453480 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:05.453445 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" podUID="f359d0e2-0a2f-4321-a1bd-c48bea8684b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 01:57:05.811114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:05.805566 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:06.177002 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:06.176942 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:06.448515 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:06.448427 5998 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 01:57:06.448515 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:06.448484 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" podUID="f359d0e2-0a2f-4321-a1bd-c48bea8684b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 01:57:07.448737 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:07.448692 5998 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 01:57:07.449253 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:07.448767 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" podUID="f359d0e2-0a2f-4321-a1bd-c48bea8684b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 01:57:07.574992 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:07.574955 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:08.448874 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:08.448832 5998 patch_prober.go:29] interesting pod/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 01:57:08.449370 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:08.448910 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" podUID="f359d0e2-0a2f-4321-a1bd-c48bea8684b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 01:57:08.705815 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:08.705670 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:09.149488 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.149408 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8"] Oct 30 01:57:09.160542 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.159862 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.161200 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.161170 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8"] Oct 30 01:57:09.173640 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.171882 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:09.175252 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.175207 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-bd098e-openshift-default-dockercfg-9lzc8\"" Oct 30 01:57:09.177540 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.177246 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jnth\" (UniqueName: \"kubernetes.io/projected/5c72e866-870f-4f28-8e32-eb303f84488d-kube-api-access-5jnth\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.177540 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.177485 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-istio-data\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.177540 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.177529 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-workload-socket\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.177729 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.177602 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c72e866-870f-4f28-8e32-eb303f84488d-istio-podinfo\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.177729 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.177645 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c72e866-870f-4f28-8e32-eb303f84488d-istiod-ca-cert\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.177729 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.177700 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-workload-certs\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.177897 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.177729 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-istio-envoy\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.177897 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.177769 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-credential-socket\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.177897 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.177794 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c72e866-870f-4f28-8e32-eb303f84488d-istio-token\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.279061 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.279021 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jnth\" (UniqueName: \"kubernetes.io/projected/5c72e866-870f-4f28-8e32-eb303f84488d-kube-api-access-5jnth\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.279269 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.279033 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:09.279269 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.279254 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-istio-data\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.279403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.279282 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-workload-socket\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.279403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.279330 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c72e866-870f-4f28-8e32-eb303f84488d-istio-podinfo\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.279403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.279363 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c72e866-870f-4f28-8e32-eb303f84488d-istiod-ca-cert\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.279563 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.279400 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-workload-certs\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.279563 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.279430 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-istio-envoy\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.279563 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.279470 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-credential-socket\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.279563 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.279494 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c72e866-870f-4f28-8e32-eb303f84488d-istio-token\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.279789 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.279763 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-workload-socket\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.279891 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.279870 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-workload-certs\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.280384 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.280360 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-credential-socket\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.280488 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.280421 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c72e866-870f-4f28-8e32-eb303f84488d-istiod-ca-cert\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.280534 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.280485 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-istio-data\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.282535 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.282508 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c72e866-870f-4f28-8e32-eb303f84488d-istio-podinfo\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.282834 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.282816 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-istio-envoy\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.303938 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.303873 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c72e866-870f-4f28-8e32-eb303f84488d-istio-token\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.306139 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.306075 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jnth\" (UniqueName: \"kubernetes.io/projected/5c72e866-870f-4f28-8e32-eb303f84488d-kube-api-access-5jnth\") pod \"e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.397795 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.397749 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:09.497113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.492403 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:09.848824 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.848660 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:09.850365 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.850275 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:09.851353 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.851329 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:09.925804 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.925771 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8"] Oct 30 01:57:09.930949 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:57:09.930917 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c72e866_870f_4f28_8e32_eb303f84488d.slice/crio-228e15568aef22b401a8146be21d33f6d0190f50bdc5b83cadc81a02ce739a8b WatchSource:0}: Error finding container 228e15568aef22b401a8146be21d33f6d0190f50bdc5b83cadc81a02ce739a8b: Status 404 returned error can't find the container with id 228e15568aef22b401a8146be21d33f6d0190f50bdc5b83cadc81a02ce739a8b Oct 30 01:57:09.934017 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.933981 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:57:09.934132 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.934114 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:57:09.934185 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:09.934167 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:57:10.181329 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:10.181289 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" event={"ID":"5c72e866-870f-4f28-8e32-eb303f84488d","Type":"ContainerStarted","Data":"5cc57826cfaa636799c44bbfe5d9b689489f4f106e10254fa7718143388ce11b"} Oct 30 01:57:10.181505 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:10.181361 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" event={"ID":"5c72e866-870f-4f28-8e32-eb303f84488d","Type":"ContainerStarted","Data":"228e15568aef22b401a8146be21d33f6d0190f50bdc5b83cadc81a02ce739a8b"} Oct 30 01:57:10.230856 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:10.230808 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" podStartSLOduration=1.230790036 podStartE2EDuration="1.230790036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:57:10.225771552 +0000 UTC m=+1659.804562632" watchObservedRunningTime="2026-10-30 01:57:10.230790036 +0000 UTC m=+1659.809581116" Oct 30 01:57:10.493569 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:10.493481 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:10.873387 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:10.873298 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:10.988431 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:10.988389 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:11.096471 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:11.096431 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:11.498643 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:11.498615 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:11.569118 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:11.557676 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:57:11.739398 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:11.739359 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:11.849357 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:11.849256 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:12.351112 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:12.350562 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:12.351665 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:12.351602 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:57:12.399977 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:12.399946 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:12.555147 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:12.555063 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:13.076711 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:13.076668 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:13.232616 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:13.232581 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:13.529105 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:13.529065 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:13.770049 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:13.770015 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:14.520909 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:14.520871 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:14.897203 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:14.897081 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:15.010537 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:15.010499 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:15.879196 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:15.879164 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:17.690363 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:17.690313 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:17.823381 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:17.823345 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:17.952851 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:17.952753 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:19.576499 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:19.576462 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:21.134147 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:21.134083 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:21.243062 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:21.243022 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:21.351183 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:21.351147 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:23.007982 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:23.007945 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:23.204573 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:23.204530 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:23.508150 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:23.508074 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:24.433799 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:24.433751 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:25.227342 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:57:25.227303 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:57:25.601213 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:25.601118 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:27.211930 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:27.211891 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:29.056830 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:29.056796 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:30.777241 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:30.777197 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:30.978219 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:30.978181 5998 scope.go:122] "RemoveContainer" containerID="462b045160cd3daa9c3777af2725fe95488ab3d61990ef24ef48963a04feab2d" Oct 30 01:57:30.988931 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:30.988909 5998 scope.go:122] "RemoveContainer" containerID="95bf7af10a3f8e92db83702c320a12e4da4997ada112d76d67e679e303ff27ba" Oct 30 01:57:30.999171 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:30.999150 5998 scope.go:122] "RemoveContainer" containerID="c1203f05c2aa1ed83b8cc1daddcec0cecb6bd218e12b99d0160229fa6237697f" Oct 30 01:57:31.011403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:31.011379 5998 scope.go:122] "RemoveContainer" containerID="d7e6c60cda2fc5088d5a126c182b5b4594bd5f20a1b4785c04066a3fedcd7458" Oct 30 01:57:31.021774 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:31.021753 5998 scope.go:122] "RemoveContainer" containerID="1a8bd0f9988c9ed2b1f6225055bb11c3758150417259b388b42ddbf57a7b1a8d" Oct 30 01:57:31.187716 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:31.187675 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:31.305970 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:31.305881 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:31.911820 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:31.911782 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:32.263562 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:32.263456 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv"] Oct 30 01:57:32.263739 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:32.263719 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" podUID="f359d0e2-0a2f-4321-a1bd-c48bea8684b1" containerName="istio-proxy" containerID="cri-o://a597da9e106ea78e072f9f7d784c7ed04f83ec63913441ba4f808778796c1ee1" gracePeriod=30 Oct 30 01:57:32.276601 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:32.276570 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf"] Oct 30 01:57:32.277472 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:32.277432 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" podUID="9d11c0bb-7924-485b-8514-1213bc06afca" containerName="istio-proxy" containerID="cri-o://9578979d4d878341adda97a7468e9354e33345c9d419bbaee3de14f5624c3850" gracePeriod=30 Oct 30 01:57:32.296475 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:32.296385 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:32.897277 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:32.897237 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:33.014111 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:33.010826 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:33.119736 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:33.119705 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:34.650681 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:34.650644 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:34.999597 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:34.999564 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:35.475733 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:35.475689 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:57:36.738052 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:36.738018 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:37.579659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.579632 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:37.633885 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.633855 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:57:37.637124 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.637063 5998 generic.go:356] "Generic (PLEG): container finished" podID="f359d0e2-0a2f-4321-a1bd-c48bea8684b1" containerID="a597da9e106ea78e072f9f7d784c7ed04f83ec63913441ba4f808778796c1ee1" exitCode=0 Oct 30 01:57:37.637285 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.637137 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" event={"ID":"f359d0e2-0a2f-4321-a1bd-c48bea8684b1","Type":"ContainerDied","Data":"a597da9e106ea78e072f9f7d784c7ed04f83ec63913441ba4f808778796c1ee1"} Oct 30 01:57:37.637285 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.637148 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" Oct 30 01:57:37.637285 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.637186 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv" event={"ID":"f359d0e2-0a2f-4321-a1bd-c48bea8684b1","Type":"ContainerDied","Data":"1014d4c0f7dfadd78f39cef612818fc69ee302a5db338f419c567fce5b2eb3ee"} Oct 30 01:57:37.637285 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.637209 5998 scope.go:122] "RemoveContainer" containerID="a597da9e106ea78e072f9f7d784c7ed04f83ec63913441ba4f808778796c1ee1" Oct 30 01:57:37.639798 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.639772 5998 generic.go:356] "Generic (PLEG): container finished" podID="9d11c0bb-7924-485b-8514-1213bc06afca" containerID="9578979d4d878341adda97a7468e9354e33345c9d419bbaee3de14f5624c3850" exitCode=0 Oct 30 01:57:37.641515 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.639810 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" event={"ID":"9d11c0bb-7924-485b-8514-1213bc06afca","Type":"ContainerDied","Data":"9578979d4d878341adda97a7468e9354e33345c9d419bbaee3de14f5624c3850"} Oct 30 01:57:37.641515 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.639855 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" event={"ID":"9d11c0bb-7924-485b-8514-1213bc06afca","Type":"ContainerDied","Data":"a4f906c22bafc1226dd709e86a6b1c4408aa0934a5793e68e36f9bdce96f1eca"} Oct 30 01:57:37.641515 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.639922 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf" Oct 30 01:57:37.650709 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.650683 5998 scope.go:122] "RemoveContainer" containerID="a597da9e106ea78e072f9f7d784c7ed04f83ec63913441ba4f808778796c1ee1" Oct 30 01:57:37.651075 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:57:37.651056 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a597da9e106ea78e072f9f7d784c7ed04f83ec63913441ba4f808778796c1ee1\": container with ID starting with a597da9e106ea78e072f9f7d784c7ed04f83ec63913441ba4f808778796c1ee1 not found: ID does not exist" containerID="a597da9e106ea78e072f9f7d784c7ed04f83ec63913441ba4f808778796c1ee1" Oct 30 01:57:37.651225 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.651100 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a597da9e106ea78e072f9f7d784c7ed04f83ec63913441ba4f808778796c1ee1"} err="failed to get container status \"a597da9e106ea78e072f9f7d784c7ed04f83ec63913441ba4f808778796c1ee1\": rpc error: code = NotFound desc = could not find container \"a597da9e106ea78e072f9f7d784c7ed04f83ec63913441ba4f808778796c1ee1\": container with ID starting with a597da9e106ea78e072f9f7d784c7ed04f83ec63913441ba4f808778796c1ee1 not found: ID does not exist" Oct 30 01:57:37.651225 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.651125 5998 scope.go:122] "RemoveContainer" containerID="9578979d4d878341adda97a7468e9354e33345c9d419bbaee3de14f5624c3850" Oct 30 01:57:37.664816 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.664717 5998 scope.go:122] "RemoveContainer" containerID="1ff026f8f2c1777effacccd9fef323c9c2ccd7c3b93bf51c04a605ff21594ed9" Oct 30 01:57:37.683114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.682997 5998 scope.go:122] "RemoveContainer" containerID="9578979d4d878341adda97a7468e9354e33345c9d419bbaee3de14f5624c3850" Oct 30 01:57:37.683393 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:57:37.683362 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9578979d4d878341adda97a7468e9354e33345c9d419bbaee3de14f5624c3850\": container with ID starting with 9578979d4d878341adda97a7468e9354e33345c9d419bbaee3de14f5624c3850 not found: ID does not exist" containerID="9578979d4d878341adda97a7468e9354e33345c9d419bbaee3de14f5624c3850" Oct 30 01:57:37.683486 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.683403 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9578979d4d878341adda97a7468e9354e33345c9d419bbaee3de14f5624c3850"} err="failed to get container status \"9578979d4d878341adda97a7468e9354e33345c9d419bbaee3de14f5624c3850\": rpc error: code = NotFound desc = could not find container \"9578979d4d878341adda97a7468e9354e33345c9d419bbaee3de14f5624c3850\": container with ID starting with 9578979d4d878341adda97a7468e9354e33345c9d419bbaee3de14f5624c3850 not found: ID does not exist" Oct 30 01:57:37.683486 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.683423 5998 scope.go:122] "RemoveContainer" containerID="1ff026f8f2c1777effacccd9fef323c9c2ccd7c3b93bf51c04a605ff21594ed9" Oct 30 01:57:37.683726 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:57:37.683706 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff026f8f2c1777effacccd9fef323c9c2ccd7c3b93bf51c04a605ff21594ed9\": container with ID starting with 1ff026f8f2c1777effacccd9fef323c9c2ccd7c3b93bf51c04a605ff21594ed9 not found: ID does not exist" containerID="1ff026f8f2c1777effacccd9fef323c9c2ccd7c3b93bf51c04a605ff21594ed9" Oct 30 01:57:37.683794 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.683732 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff026f8f2c1777effacccd9fef323c9c2ccd7c3b93bf51c04a605ff21594ed9"} err="failed to get container status \"1ff026f8f2c1777effacccd9fef323c9c2ccd7c3b93bf51c04a605ff21594ed9\": rpc error: code = NotFound desc = could not find container \"1ff026f8f2c1777effacccd9fef323c9c2ccd7c3b93bf51c04a605ff21594ed9\": container with ID starting with 1ff026f8f2c1777effacccd9fef323c9c2ccd7c3b93bf51c04a605ff21594ed9 not found: ID does not exist" Oct 30 01:57:37.734061 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.734013 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-workload-certs\") pod \"9d11c0bb-7924-485b-8514-1213bc06afca\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " Oct 30 01:57:37.734250 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.734183 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-credential-socket\") pod \"9d11c0bb-7924-485b-8514-1213bc06afca\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " Oct 30 01:57:37.734314 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.734277 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-workload-socket\") pod \"9d11c0bb-7924-485b-8514-1213bc06afca\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " Oct 30 01:57:37.734370 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.734324 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d11c0bb-7924-485b-8514-1213bc06afca-kube-api-access-k2bc7\" (UniqueName: \"kubernetes.io/projected/9d11c0bb-7924-485b-8514-1213bc06afca-kube-api-access-k2bc7\") pod \"9d11c0bb-7924-485b-8514-1213bc06afca\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " Oct 30 01:57:37.734370 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.734337 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-workload-certs" pod "9d11c0bb-7924-485b-8514-1213bc06afca" (UID: "9d11c0bb-7924-485b-8514-1213bc06afca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.734485 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.734389 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-token\" (UniqueName: \"kubernetes.io/projected/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-token\") pod \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " Oct 30 01:57:37.734485 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.734456 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d11c0bb-7924-485b-8514-1213bc06afca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d11c0bb-7924-485b-8514-1213bc06afca-istiod-ca-cert\") pod \"9d11c0bb-7924-485b-8514-1213bc06afca\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " Oct 30 01:57:37.734589 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.734484 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d11c0bb-7924-485b-8514-1213bc06afca-istio-token\" (UniqueName: \"kubernetes.io/projected/9d11c0bb-7924-485b-8514-1213bc06afca-istio-token\") pod \"9d11c0bb-7924-485b-8514-1213bc06afca\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " Oct 30 01:57:37.734589 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.734535 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-workload-socket" pod "9d11c0bb-7924-485b-8514-1213bc06afca" (UID: "9d11c0bb-7924-485b-8514-1213bc06afca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.735160 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.734875 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-credential-socket" pod "9d11c0bb-7924-485b-8514-1213bc06afca" (UID: "9d11c0bb-7924-485b-8514-1213bc06afca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.735160 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.734912 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d11c0bb-7924-485b-8514-1213bc06afca-istiod-ca-cert" pod "9d11c0bb-7924-485b-8514-1213bc06afca" (UID: "9d11c0bb-7924-485b-8514-1213bc06afca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:57:37.735160 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.734522 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-podinfo\") pod \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " Oct 30 01:57:37.735160 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.735010 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-workload-socket\") pod \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " Oct 30 01:57:37.735476 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.735353 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-workload-socket" pod "f359d0e2-0a2f-4321-a1bd-c48bea8684b1" (UID: "f359d0e2-0a2f-4321-a1bd-c48bea8684b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.735476 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.735403 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d11c0bb-7924-485b-8514-1213bc06afca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d11c0bb-7924-485b-8514-1213bc06afca-istio-podinfo\") pod \"9d11c0bb-7924-485b-8514-1213bc06afca\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " Oct 30 01:57:37.735476 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.735459 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-kube-api-access-x7bwq\" (UniqueName: \"kubernetes.io/projected/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-kube-api-access-x7bwq\") pod \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " Oct 30 01:57:37.735810 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.735790 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-istio-envoy\") pod \"9d11c0bb-7924-485b-8514-1213bc06afca\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " Oct 30 01:57:37.736289 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.735974 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-istio-data\") pod \"9d11c0bb-7924-485b-8514-1213bc06afca\" (UID: \"9d11c0bb-7924-485b-8514-1213bc06afca\") " Oct 30 01:57:37.736289 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.736248 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-istio-data" pod "9d11c0bb-7924-485b-8514-1213bc06afca" (UID: "9d11c0bb-7924-485b-8514-1213bc06afca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.736463 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.736312 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-credential-socket\") pod \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " Oct 30 01:57:37.736552 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.736531 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-credential-socket" pod "f359d0e2-0a2f-4321-a1bd-c48bea8684b1" (UID: "f359d0e2-0a2f-4321-a1bd-c48bea8684b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.736611 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.736586 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istiod-ca-cert\") pod \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " Oct 30 01:57:37.736991 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.736941 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istiod-ca-cert" pod "f359d0e2-0a2f-4321-a1bd-c48bea8684b1" (UID: "f359d0e2-0a2f-4321-a1bd-c48bea8684b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:57:37.737198 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.737080 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-data" pod "f359d0e2-0a2f-4321-a1bd-c48bea8684b1" (UID: "f359d0e2-0a2f-4321-a1bd-c48bea8684b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.737289 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.736617 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-data\") pod \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " Oct 30 01:57:37.737401 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.737323 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-envoy\") pod \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " Oct 30 01:57:37.737586 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.737560 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d11c0bb-7924-485b-8514-1213bc06afca-istio-token" pod "9d11c0bb-7924-485b-8514-1213bc06afca" (UID: "9d11c0bb-7924-485b-8514-1213bc06afca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:37.737662 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.737647 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-workload-certs\") pod \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\" (UID: \"f359d0e2-0a2f-4321-a1bd-c48bea8684b1\") " Oct 30 01:57:37.737739 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.737717 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d11c0bb-7924-485b-8514-1213bc06afca-kube-api-access-k2bc7" pod "9d11c0bb-7924-485b-8514-1213bc06afca" (UID: "9d11c0bb-7924-485b-8514-1213bc06afca"). InnerVolumeSpecName "kube-api-access-k2bc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:37.737946 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.737905 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-workload-certs" pod "f359d0e2-0a2f-4321-a1bd-c48bea8684b1" (UID: "f359d0e2-0a2f-4321-a1bd-c48bea8684b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.738935 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.738915 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d11c0bb-7924-485b-8514-1213bc06afca-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.739447 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.739428 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d11c0bb-7924-485b-8514-1213bc06afca-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.739592 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.739579 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.739727 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.739714 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.739845 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.739824 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.739845 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.739844 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.740000 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.739860 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.740000 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.739873 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.740000 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.739356 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-kube-api-access-x7bwq" pod "f359d0e2-0a2f-4321-a1bd-c48bea8684b1" (UID: "f359d0e2-0a2f-4321-a1bd-c48bea8684b1"). InnerVolumeSpecName "kube-api-access-x7bwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:37.740000 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.739886 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.740000 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.739722 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-istio-envoy" pod "9d11c0bb-7924-485b-8514-1213bc06afca" (UID: "9d11c0bb-7924-485b-8514-1213bc06afca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.740000 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.739907 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.740000 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.739921 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.740000 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.739936 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2bc7\" (UniqueName: \"kubernetes.io/projected/9d11c0bb-7924-485b-8514-1213bc06afca-kube-api-access-k2bc7\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.740411 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.740222 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-envoy" pod "f359d0e2-0a2f-4321-a1bd-c48bea8684b1" (UID: "f359d0e2-0a2f-4321-a1bd-c48bea8684b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:57:37.740411 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.740289 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-podinfo" pod "f359d0e2-0a2f-4321-a1bd-c48bea8684b1" (UID: "f359d0e2-0a2f-4321-a1bd-c48bea8684b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:57:37.740411 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.740298 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-token" pod "f359d0e2-0a2f-4321-a1bd-c48bea8684b1" (UID: "f359d0e2-0a2f-4321-a1bd-c48bea8684b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:37.741490 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.741468 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d11c0bb-7924-485b-8514-1213bc06afca-istio-podinfo" pod "9d11c0bb-7924-485b-8514-1213bc06afca" (UID: "9d11c0bb-7924-485b-8514-1213bc06afca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:57:37.841773 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.841669 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.841773 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.841706 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.841773 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.841719 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.841773 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.841730 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d11c0bb-7924-485b-8514-1213bc06afca-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.841773 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.841744 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7bwq\" (UniqueName: \"kubernetes.io/projected/f359d0e2-0a2f-4321-a1bd-c48bea8684b1-kube-api-access-x7bwq\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.841773 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.841757 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d11c0bb-7924-485b-8514-1213bc06afca-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:37.991922 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:37.991873 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv"] Oct 30 01:57:38.003241 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:38.003203 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-tqrlv"] Oct 30 01:57:38.035809 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:38.035771 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf"] Oct 30 01:57:38.052496 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:38.052454 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-922b2c-openshift-default-64b4cfc669-5brlf"] Oct 30 01:57:38.313361 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:38.313328 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:38.423517 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:38.423481 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:38.535053 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:38.535008 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:38.539848 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:38.539814 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d11c0bb-7924-485b-8514-1213bc06afca" path="/var/lib/kubelet/pods/9d11c0bb-7924-485b-8514-1213bc06afca/volumes" Oct 30 01:57:38.540480 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:38.540459 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f359d0e2-0a2f-4321-a1bd-c48bea8684b1" path="/var/lib/kubelet/pods/f359d0e2-0a2f-4321-a1bd-c48bea8684b1/volumes" Oct 30 01:57:38.948767 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:38.948731 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:39.060475 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:39.060414 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:40.736656 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:40.736609 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:41.036790 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:41.036706 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x"] Oct 30 01:57:41.048483 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:41.048449 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f0cc884-dm48x"] Oct 30 01:57:42.265871 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:42.265831 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:42.542001 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:42.541918 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aac29fe5-9261-48a6-9b20-d9a192087501" path="/var/lib/kubelet/pods/aac29fe5-9261-48a6-9b20-d9a192087501/volumes" Oct 30 01:57:42.566994 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:42.566955 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:44.328620 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:44.328581 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:44.632816 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:44.632715 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:45.955375 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:45.955331 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:47.136644 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:47.136613 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:47.589334 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:47.589299 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:48.238280 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:48.238242 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:48.979004 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:48.978969 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11184-24dcbd"] Oct 30 01:57:48.979545 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:48.979531 5998 cpu_manager.go:414] "containerMap: removing container" podUID="f359d0e2-0a2f-4321-a1bd-c48bea8684b1" containerName="istio-proxy" Oct 30 01:57:48.979600 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:48.979547 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f359d0e2-0a2f-4321-a1bd-c48bea8684b1" containerName="istio-proxy" Oct 30 01:57:48.979600 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:48.979570 5998 cpu_manager.go:414] "containerMap: removing container" podUID="9d11c0bb-7924-485b-8514-1213bc06afca" containerName="istio-proxy" Oct 30 01:57:48.979600 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:48.979576 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d11c0bb-7924-485b-8514-1213bc06afca" containerName="istio-proxy" Oct 30 01:57:48.979600 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:48.979588 5998 cpu_manager.go:414] "containerMap: removing container" podUID="9d11c0bb-7924-485b-8514-1213bc06afca" containerName="istio-proxy" Oct 30 01:57:48.979600 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:48.979594 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d11c0bb-7924-485b-8514-1213bc06afca" containerName="istio-proxy" Oct 30 01:57:48.979750 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:48.979688 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="f359d0e2-0a2f-4321-a1bd-c48bea8684b1" containerName="istio-proxy" Oct 30 01:57:48.979750 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:48.979698 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d11c0bb-7924-485b-8514-1213bc06afca" containerName="istio-proxy" Oct 30 01:57:48.979750 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:48.979705 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d11c0bb-7924-485b-8514-1213bc06afca" containerName="istio-proxy" Oct 30 01:57:48.985015 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:48.984988 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11184-24dcbd" Oct 30 01:57:48.993834 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:48.993808 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:49.009263 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:49.009238 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2sddb\"" Oct 30 01:57:49.015523 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:49.015490 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11184-24dcbd"] Oct 30 01:57:49.085343 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:49.085303 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtt6q\" (UniqueName: \"kubernetes.io/projected/073dbf31-5841-46f7-ac58-ce43ef76286f-kube-api-access-wtt6q\") pod \"test-curl-11184-24dcbd\" (UID: \"073dbf31-5841-46f7-ac58-ce43ef76286f\") " pod="opendatahub/test-curl-11184-24dcbd" Oct 30 01:57:49.186663 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:49.186624 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtt6q\" (UniqueName: \"kubernetes.io/projected/073dbf31-5841-46f7-ac58-ce43ef76286f-kube-api-access-wtt6q\") pod \"test-curl-11184-24dcbd\" (UID: \"073dbf31-5841-46f7-ac58-ce43ef76286f\") " pod="opendatahub/test-curl-11184-24dcbd" Oct 30 01:57:49.229798 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:49.229716 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtt6q\" (UniqueName: \"kubernetes.io/projected/073dbf31-5841-46f7-ac58-ce43ef76286f-kube-api-access-wtt6q\") pod \"test-curl-11184-24dcbd\" (UID: \"073dbf31-5841-46f7-ac58-ce43ef76286f\") " pod="opendatahub/test-curl-11184-24dcbd" Oct 30 01:57:49.231379 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:49.231348 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:49.298156 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:49.298124 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11184-24dcbd" Oct 30 01:57:49.671730 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:49.671703 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11184-24dcbd"] Oct 30 01:57:49.672764 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:57:49.672742 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod073dbf31_5841_46f7_ac58_ce43ef76286f.slice/crio-d0a190a70317f2062ec220e1e9fe814c10b05563021c2ba7586766bb40fdcb87 WatchSource:0}: Error finding container d0a190a70317f2062ec220e1e9fe814c10b05563021c2ba7586766bb40fdcb87: Status 404 returned error can't find the container with id d0a190a70317f2062ec220e1e9fe814c10b05563021c2ba7586766bb40fdcb87 Oct 30 01:57:49.674788 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:49.674769 5998 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:57:49.784462 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:49.784426 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11184-24dcbd" event={"ID":"073dbf31-5841-46f7-ac58-ce43ef76286f","Type":"ContainerStarted","Data":"d0a190a70317f2062ec220e1e9fe814c10b05563021c2ba7586766bb40fdcb87"} Oct 30 01:57:50.782167 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:50.782128 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:50.795219 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:50.795182 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11184-24dcbd" event={"ID":"073dbf31-5841-46f7-ac58-ce43ef76286f","Type":"ContainerStarted","Data":"10f931701bd235f9d0ede5df3a1b5036d5c6961ae43370c3420eea01b3aa1767"} Oct 30 01:57:50.833748 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:50.833689 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11184-24dcbd" podStartSLOduration=2.190485309 podStartE2EDuration="2.833673285s" totalImagesPullingTime="643.187976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:57:50.831993596 +0000 UTC m=+1700.410784678" watchObservedRunningTime="2026-10-30 01:57:50.833673285 +0000 UTC m=+1700.412464431" Oct 30 01:57:51.769320 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:51.769281 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11184-24dcbd"] Oct 30 01:57:51.773988 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:51.773959 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11184-24dcbd"] Oct 30 01:57:51.968740 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:51.968706 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:52.033118 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:52.033029 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8"] Oct 30 01:57:52.045475 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:52.045441 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca5ed0bf-bzhp8"] Oct 30 01:57:52.310563 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:52.310486 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:52.482850 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:52.482809 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:52.576017 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:52.575934 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32f8b0ef-77f6-441e-ac51-a5e6fccbb750" path="/var/lib/kubelet/pods/32f8b0ef-77f6-441e-ac51-a5e6fccbb750/volumes" Oct 30 01:57:52.816021 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:52.815977 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11184-24dcbd" podUID="073dbf31-5841-46f7-ac58-ce43ef76286f" containerName="test-curl-11184-24dcbd" containerID="cri-o://10f931701bd235f9d0ede5df3a1b5036d5c6961ae43370c3420eea01b3aa1767" gracePeriod=2 Oct 30 01:57:53.419209 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:53.419155 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:53.531143 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:53.531104 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:53.824877 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:53.824785 5998 generic.go:356] "Generic (PLEG): container finished" podID="fa5cf05f-5444-459e-ba19-657e8c9f4ee1" containerID="e800b0c096f46733333b293bfa3c5d29fa9c7af0475ef21250822a59d77bd070" exitCode=0 Oct 30 01:57:53.824877 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:53.824862 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" event={"ID":"fa5cf05f-5444-459e-ba19-657e8c9f4ee1","Type":"ContainerDied","Data":"e800b0c096f46733333b293bfa3c5d29fa9c7af0475ef21250822a59d77bd070"} Oct 30 01:57:53.825392 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:53.825373 5998 scope.go:122] "RemoveContainer" containerID="e800b0c096f46733333b293bfa3c5d29fa9c7af0475ef21250822a59d77bd070" Oct 30 01:57:53.827049 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:53.827014 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:57:53.827215 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:53.827117 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:57:53.827215 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:53.827160 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:57:54.031798 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:54.031744 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:54.847683 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:54.847648 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" event={"ID":"fa5cf05f-5444-459e-ba19-657e8c9f4ee1","Type":"ContainerStarted","Data":"4c84ea911af364487e52602de6b1278ce8bd344706e4e06a4a5d9f0201414449"} Oct 30 01:57:55.096583 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:55.096557 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11184-24dcbd" Oct 30 01:57:55.190845 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:55.190807 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:55.272339 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:57:55.272289 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:57:55.275327 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:55.275299 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/073dbf31-5841-46f7-ac58-ce43ef76286f-kube-api-access-wtt6q\" (UniqueName: \"kubernetes.io/projected/073dbf31-5841-46f7-ac58-ce43ef76286f-kube-api-access-wtt6q\") pod \"073dbf31-5841-46f7-ac58-ce43ef76286f\" (UID: \"073dbf31-5841-46f7-ac58-ce43ef76286f\") " Oct 30 01:57:55.280333 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:55.280295 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/073dbf31-5841-46f7-ac58-ce43ef76286f-kube-api-access-wtt6q" pod "073dbf31-5841-46f7-ac58-ce43ef76286f" (UID: "073dbf31-5841-46f7-ac58-ce43ef76286f"). InnerVolumeSpecName "kube-api-access-wtt6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:57:55.378071 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:55.377976 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtt6q\" (UniqueName: \"kubernetes.io/projected/073dbf31-5841-46f7-ac58-ce43ef76286f-kube-api-access-wtt6q\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:57:55.575417 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:55.575379 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:57:55.576461 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:55.576434 5998 patch_prober.go:29] interesting pod/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:57:55.576581 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:55.576481 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" podUID="fa5cf05f-5444-459e-ba19-657e8c9f4ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 01:57:55.769558 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:55.769514 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:55.856674 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:55.856640 5998 generic.go:356] "Generic (PLEG): container finished" podID="073dbf31-5841-46f7-ac58-ce43ef76286f" containerID="10f931701bd235f9d0ede5df3a1b5036d5c6961ae43370c3420eea01b3aa1767" exitCode=137 Oct 30 01:57:55.857147 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:55.856698 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11184-24dcbd" Oct 30 01:57:55.857147 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:55.856702 5998 scope.go:122] "RemoveContainer" containerID="10f931701bd235f9d0ede5df3a1b5036d5c6961ae43370c3420eea01b3aa1767" Oct 30 01:57:55.868989 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:55.868788 5998 scope.go:122] "RemoveContainer" containerID="10f931701bd235f9d0ede5df3a1b5036d5c6961ae43370c3420eea01b3aa1767" Oct 30 01:57:55.869143 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:57:55.869117 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10f931701bd235f9d0ede5df3a1b5036d5c6961ae43370c3420eea01b3aa1767\": container with ID starting with 10f931701bd235f9d0ede5df3a1b5036d5c6961ae43370c3420eea01b3aa1767 not found: ID does not exist" containerID="10f931701bd235f9d0ede5df3a1b5036d5c6961ae43370c3420eea01b3aa1767" Oct 30 01:57:55.869221 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:55.869153 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10f931701bd235f9d0ede5df3a1b5036d5c6961ae43370c3420eea01b3aa1767"} err="failed to get container status \"10f931701bd235f9d0ede5df3a1b5036d5c6961ae43370c3420eea01b3aa1767\": rpc error: code = NotFound desc = could not find container \"10f931701bd235f9d0ede5df3a1b5036d5c6961ae43370c3420eea01b3aa1767\": container with ID starting with 10f931701bd235f9d0ede5df3a1b5036d5c6961ae43370c3420eea01b3aa1767 not found: ID does not exist" Oct 30 01:57:55.879461 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:57:55.879431 5998 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11184-24dcbd\" is forbidden: User \"system:node:ip-10-0-130-0.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-0.ec2.internal' and this object" podUID="073dbf31-5841-46f7-ac58-ce43ef76286f" pod="opendatahub/test-curl-11184-24dcbd" Oct 30 01:57:56.032281 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:56.032197 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8"] Oct 30 01:57:56.043562 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:56.043534 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df4b9a47-57ff8"] Oct 30 01:57:56.209983 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:56.209910 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:56.328633 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:56.328533 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:56.539048 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:56.539017 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="073dbf31-5841-46f7-ac58-ce43ef76286f" path="/var/lib/kubelet/pods/073dbf31-5841-46f7-ac58-ce43ef76286f/volumes" Oct 30 01:57:56.539529 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:56.539514 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb8b7b91-c3df-4b25-9eb0-02dad493b607" path="/var/lib/kubelet/pods/fb8b7b91-c3df-4b25-9eb0-02dad493b607/volumes" Oct 30 01:57:56.576191 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:56.576161 5998 patch_prober.go:29] interesting pod/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:57:56.576330 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:56.576211 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" podUID="fa5cf05f-5444-459e-ba19-657e8c9f4ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 01:57:57.157043 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:57.157002 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:57.263980 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:57.263945 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8"] Oct 30 01:57:57.324690 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:57.324597 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:57.460771 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:57.460732 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:57.594710 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:57.594667 5998 patch_prober.go:29] interesting pod/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:57:57.594909 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:57.594746 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" podUID="fa5cf05f-5444-459e-ba19-657e8c9f4ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 01:57:58.377659 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:58.377621 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:58.569295 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:58.569263 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:58.576844 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:58.576802 5998 patch_prober.go:29] interesting pod/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:57:58.577020 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:58.576873 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" podUID="fa5cf05f-5444-459e-ba19-657e8c9f4ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 01:57:58.707609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:58.707571 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:58.890511 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:58.890471 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:59.104201 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:59.100361 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:59.213114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:59.212326 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:57:59.575869 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:59.575826 5998 patch_prober.go:29] interesting pod/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 01:57:59.576417 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:59.575897 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" podUID="fa5cf05f-5444-459e-ba19-657e8c9f4ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 01:57:59.924499 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:57:59.924469 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:58:00.580263 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:00.580233 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:58:00.862487 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:00.862382 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:00.922643 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:00.922611 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:58:02.442636 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:02.442600 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:02.566292 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:02.566254 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:02.674303 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:02.674270 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:02.864051 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:02.863963 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:04.172450 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:04.172415 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:04.330850 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:04.330796 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:04.582069 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:04.581938 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:05.972137 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:05.972056 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:06.033610 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:06.033574 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8"] Oct 30 01:58:06.042969 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:06.042930 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0719b37e-74rj8"] Oct 30 01:58:06.548410 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:06.548373 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf8bba3-cf02-49be-a66c-ff95056bbc12" path="/var/lib/kubelet/pods/fdf8bba3-cf02-49be-a66c-ff95056bbc12/volumes" Oct 30 01:58:06.976940 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:06.976900 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:08.510428 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:08.510382 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:58:08.796205 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:08.796119 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:10.042509 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:10.042475 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w"] Oct 30 01:58:10.053693 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:10.053661 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4875c84f-p7z7w"] Oct 30 01:58:10.547345 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:10.547310 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5318fb38-6f90-4cf4-8d8a-597de8ee1a50" path="/var/lib/kubelet/pods/5318fb38-6f90-4cf4-8d8a-597de8ee1a50/volumes" Oct 30 01:58:10.755483 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:10.755438 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:10.954899 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:10.954862 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:11.633640 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:11.633602 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:11.938844 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:11.938792 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:12.284813 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:12.284709 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" podUID="cb036aa6-48d5-465a-a4c8-5860ce3d69eb" containerName="main" containerID="cri-o://e9b32b219086d8e69c0615c448d5e40b7e36d11fb0df3368ac70704e6cd8d4f3" gracePeriod=45 Oct 30 01:58:12.455980 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:12.455904 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:12.650172 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:12.617003 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:13.019080 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.019042 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:13.040607 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.040575 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:58:13.099559 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.099504 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-tmp-dir\") pod \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " Oct 30 01:58:13.099891 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.099864 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-model-cache\") pod \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " Oct 30 01:58:13.104192 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.100172 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-dshm\") pod \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " Oct 30 01:58:13.104445 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.104420 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-kube-api-access-ds4jl\" (UniqueName: \"kubernetes.io/projected/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-kube-api-access-ds4jl\") pod \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " Oct 30 01:58:13.107914 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.107882 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-home\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-home\") pod \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " Oct 30 01:58:13.123121 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.114274 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-tls-certs\") pod \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\" (UID: \"cb036aa6-48d5-465a-a4c8-5860ce3d69eb\") " Oct 30 01:58:13.162655 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.158351 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-tmp-dir" pod "cb036aa6-48d5-465a-a4c8-5860ce3d69eb" (UID: "cb036aa6-48d5-465a-a4c8-5860ce3d69eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:13.162655 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.162189 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-home" pod "cb036aa6-48d5-465a-a4c8-5860ce3d69eb" (UID: "cb036aa6-48d5-465a-a4c8-5860ce3d69eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:13.162655 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.162404 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-model-cache" pod "cb036aa6-48d5-465a-a4c8-5860ce3d69eb" (UID: "cb036aa6-48d5-465a-a4c8-5860ce3d69eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:13.162655 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.162577 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-kube-api-access-ds4jl" pod "cb036aa6-48d5-465a-a4c8-5860ce3d69eb" (UID: "cb036aa6-48d5-465a-a4c8-5860ce3d69eb"). InnerVolumeSpecName "kube-api-access-ds4jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:13.174042 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.173430 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-tls-certs" pod "cb036aa6-48d5-465a-a4c8-5860ce3d69eb" (UID: "cb036aa6-48d5-465a-a4c8-5860ce3d69eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:58:13.175953 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.175557 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-dshm" pod "cb036aa6-48d5-465a-a4c8-5860ce3d69eb" (UID: "cb036aa6-48d5-465a-a4c8-5860ce3d69eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:13.180977 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.178410 5998 generic.go:356] "Generic (PLEG): container finished" podID="cb036aa6-48d5-465a-a4c8-5860ce3d69eb" containerID="e9b32b219086d8e69c0615c448d5e40b7e36d11fb0df3368ac70704e6cd8d4f3" exitCode=0 Oct 30 01:58:13.180977 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.178495 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" event={"ID":"cb036aa6-48d5-465a-a4c8-5860ce3d69eb","Type":"ContainerDied","Data":"e9b32b219086d8e69c0615c448d5e40b7e36d11fb0df3368ac70704e6cd8d4f3"} Oct 30 01:58:13.180977 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.178532 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" event={"ID":"cb036aa6-48d5-465a-a4c8-5860ce3d69eb","Type":"ContainerDied","Data":"c80e32247fb93e3d164e18694fbaf5f22eb5500bad1666c7140ed895008a4cb1"} Oct 30 01:58:13.180977 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.178554 5998 scope.go:122] "RemoveContainer" containerID="e9b32b219086d8e69c0615c448d5e40b7e36d11fb0df3368ac70704e6cd8d4f3" Oct 30 01:58:13.180977 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.178710 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8" Oct 30 01:58:13.199347 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.199316 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp"] Oct 30 01:58:13.264456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.251526 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds4jl\" (UniqueName: \"kubernetes.io/projected/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-kube-api-access-ds4jl\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:58:13.264456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.251567 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:58:13.264456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.251583 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:58:13.264456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.251597 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:58:13.264456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.251611 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:58:13.264456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.251624 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb036aa6-48d5-465a-a4c8-5860ce3d69eb-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:58:13.264456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.259988 5998 cpu_manager.go:414] "containerMap: removing container" podUID="cb036aa6-48d5-465a-a4c8-5860ce3d69eb" containerName="main" Oct 30 01:58:13.264456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.260010 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb036aa6-48d5-465a-a4c8-5860ce3d69eb" containerName="main" Oct 30 01:58:13.264456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.260029 5998 cpu_manager.go:414] "containerMap: removing container" podUID="073dbf31-5841-46f7-ac58-ce43ef76286f" containerName="test-curl-11184-24dcbd" Oct 30 01:58:13.264456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.260037 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="073dbf31-5841-46f7-ac58-ce43ef76286f" containerName="test-curl-11184-24dcbd" Oct 30 01:58:13.264456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.260573 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb036aa6-48d5-465a-a4c8-5860ce3d69eb" containerName="main" Oct 30 01:58:13.264456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.260593 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="073dbf31-5841-46f7-ac58-ce43ef76286f" containerName="test-curl-11184-24dcbd" Oct 30 01:58:13.273483 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.273296 5998 scope.go:122] "RemoveContainer" containerID="e9b32b219086d8e69c0615c448d5e40b7e36d11fb0df3368ac70704e6cd8d4f3" Oct 30 01:58:13.274567 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:58:13.274437 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b32b219086d8e69c0615c448d5e40b7e36d11fb0df3368ac70704e6cd8d4f3\": container with ID starting with e9b32b219086d8e69c0615c448d5e40b7e36d11fb0df3368ac70704e6cd8d4f3 not found: ID does not exist" containerID="e9b32b219086d8e69c0615c448d5e40b7e36d11fb0df3368ac70704e6cd8d4f3" Oct 30 01:58:13.274567 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.274487 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b32b219086d8e69c0615c448d5e40b7e36d11fb0df3368ac70704e6cd8d4f3"} err="failed to get container status \"e9b32b219086d8e69c0615c448d5e40b7e36d11fb0df3368ac70704e6cd8d4f3\": rpc error: code = NotFound desc = could not find container \"e9b32b219086d8e69c0615c448d5e40b7e36d11fb0df3368ac70704e6cd8d4f3\": container with ID starting with e9b32b219086d8e69c0615c448d5e40b7e36d11fb0df3368ac70704e6cd8d4f3 not found: ID does not exist" Oct 30 01:58:13.286728 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.286702 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp"] Oct 30 01:58:13.286840 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.286817 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp" Oct 30 01:58:13.297611 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.297584 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8"] Oct 30 01:58:13.310728 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.310155 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bd098e\"" Oct 30 01:58:13.310728 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.310296 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bd098e-dockercfg-8skhf\"" Oct 30 01:58:13.337898 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.337861 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2933ff00/e2e-lifecycle-model-2933ff00-kserve-69ddc64cc-ds9q8"] Oct 30 01:58:13.352589 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.352561 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9901c539-93e6-407d-b99c-b7cd2ad64356-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp\" (UID: \"9901c539-93e6-407d-b99c-b7cd2ad64356\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp" Oct 30 01:58:13.352735 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.352604 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdxf6\" (UniqueName: \"kubernetes.io/projected/9901c539-93e6-407d-b99c-b7cd2ad64356-kube-api-access-bdxf6\") pod \"payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp\" (UID: \"9901c539-93e6-407d-b99c-b7cd2ad64356\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp" Oct 30 01:58:13.454668 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.454626 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9901c539-93e6-407d-b99c-b7cd2ad64356-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp\" (UID: \"9901c539-93e6-407d-b99c-b7cd2ad64356\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp" Oct 30 01:58:13.454869 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.454685 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdxf6\" (UniqueName: \"kubernetes.io/projected/9901c539-93e6-407d-b99c-b7cd2ad64356-kube-api-access-bdxf6\") pod \"payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp\" (UID: \"9901c539-93e6-407d-b99c-b7cd2ad64356\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp" Oct 30 01:58:13.455534 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.455509 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9901c539-93e6-407d-b99c-b7cd2ad64356-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp\" (UID: \"9901c539-93e6-407d-b99c-b7cd2ad64356\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp" Oct 30 01:58:13.480507 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.480472 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdxf6\" (UniqueName: \"kubernetes.io/projected/9901c539-93e6-407d-b99c-b7cd2ad64356-kube-api-access-bdxf6\") pod \"payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp\" (UID: \"9901c539-93e6-407d-b99c-b7cd2ad64356\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp" Oct 30 01:58:13.643789 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.643686 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp" Oct 30 01:58:13.695957 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.695815 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:13.966035 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:13.966009 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp"] Oct 30 01:58:13.967810 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:58:13.967782 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9901c539_93e6_407d_b99c_b7cd2ad64356.slice/crio-57d8b328ae9fc27ea6b14dda13ac2693b6f2969bb893d49857e130d227d6e103 WatchSource:0}: Error finding container 57d8b328ae9fc27ea6b14dda13ac2693b6f2969bb893d49857e130d227d6e103: Status 404 returned error can't find the container with id 57d8b328ae9fc27ea6b14dda13ac2693b6f2969bb893d49857e130d227d6e103 Oct 30 01:58:14.189547 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:14.189506 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp" event={"ID":"9901c539-93e6-407d-b99c-b7cd2ad64356","Type":"ContainerStarted","Data":"57d8b328ae9fc27ea6b14dda13ac2693b6f2969bb893d49857e130d227d6e103"} Oct 30 01:58:14.545014 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:14.543902 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb036aa6-48d5-465a-a4c8-5860ce3d69eb" path="/var/lib/kubelet/pods/cb036aa6-48d5-465a-a4c8-5860ce3d69eb/volumes" Oct 30 01:58:15.377264 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:15.377226 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:16.259709 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:16.259666 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp" event={"ID":"9901c539-93e6-407d-b99c-b7cd2ad64356","Type":"ContainerStarted","Data":"c2ab70ea125c238febfa054866f8d33f9e25142e0a8e19cd7edef46a42b75463"} Oct 30 01:58:16.260916 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:16.260892 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp" Oct 30 01:58:16.307172 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:16.307076 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp" podStartSLOduration=1.82079106 podStartE2EDuration="3.307054293s" totalImagesPullingTime="1.486263233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:16.304506623 +0000 UTC m=+1725.883297705" watchObservedRunningTime="2026-10-30 01:58:16.307054293 +0000 UTC m=+1725.885845375" Oct 30 01:58:16.971222 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:16.971183 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:18.564296 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:18.564252 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:19.293698 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:19.293589 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:19.306440 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:19.306409 5998 generic.go:356] "Generic (PLEG): container finished" podID="5c72e866-870f-4f28-8e32-eb303f84488d" containerID="5cc57826cfaa636799c44bbfe5d9b689489f4f106e10254fa7718143388ce11b" exitCode=0 Oct 30 01:58:19.306730 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:19.306680 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" event={"ID":"5c72e866-870f-4f28-8e32-eb303f84488d","Type":"ContainerDied","Data":"5cc57826cfaa636799c44bbfe5d9b689489f4f106e10254fa7718143388ce11b"} Oct 30 01:58:19.307353 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:19.307331 5998 scope.go:122] "RemoveContainer" containerID="5cc57826cfaa636799c44bbfe5d9b689489f4f106e10254fa7718143388ce11b" Oct 30 01:58:19.309124 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:19.309064 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:58:19.309220 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:19.309179 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:58:19.309260 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:19.309229 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:58:19.411781 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:19.411749 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:19.945119 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:19.937578 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:20.329175 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:20.328914 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" event={"ID":"5c72e866-870f-4f28-8e32-eb303f84488d","Type":"ContainerStarted","Data":"e13191c869b4297cc7184685ff51cef1d7a02c8c25a7dabdb89a892b66699d2e"} Oct 30 01:58:20.456635 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:20.456594 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:20.493335 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:20.493302 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:58:20.493505 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:20.493472 5998 patch_prober.go:29] interesting pod/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 01:58:20.493549 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:20.493515 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" podUID="5c72e866-870f-4f28-8e32-eb303f84488d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 01:58:20.567808 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:20.567758 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:21.493862 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:21.493783 5998 patch_prober.go:29] interesting pod/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 01:58:21.493862 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:21.493851 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" podUID="5c72e866-870f-4f28-8e32-eb303f84488d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 01:58:22.037899 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:22.037865 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd"] Oct 30 01:58:22.048501 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:22.048472 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bd2b5f4-68hrd"] Oct 30 01:58:22.334618 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:22.334535 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:22.493427 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:22.493387 5998 patch_prober.go:29] interesting pod/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 01:58:22.493663 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:22.493462 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" podUID="5c72e866-870f-4f28-8e32-eb303f84488d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 01:58:22.547191 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:22.547152 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82965b88-ae62-48bb-896c-6089b43707a2" path="/var/lib/kubelet/pods/82965b88-ae62-48bb-896c-6089b43707a2/volumes" Oct 30 01:58:23.493591 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:23.493554 5998 patch_prober.go:29] interesting pod/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 01:58:23.493774 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:23.493609 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" podUID="5c72e866-870f-4f28-8e32-eb303f84488d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 01:58:24.092443 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:24.092407 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:24.493797 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:24.493760 5998 patch_prober.go:29] interesting pod/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 01:58:24.493983 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:24.493831 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" podUID="5c72e866-870f-4f28-8e32-eb303f84488d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 01:58:25.035205 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:25.035167 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc"] Oct 30 01:58:25.037015 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:25.036965 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a20ee2e-rg2lc"] Oct 30 01:58:25.246106 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:58:25.246057 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:58:25.494013 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:25.493971 5998 patch_prober.go:29] interesting pod/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 01:58:25.494233 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:25.494056 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" podUID="5c72e866-870f-4f28-8e32-eb303f84488d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 01:58:25.680735 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:25.680703 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:26.498160 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:26.498130 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:58:26.498653 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:26.498579 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:58:26.499403 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:26.499384 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 01:58:26.540690 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:26.540655 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="980a30c0-fdfa-4d75-906e-c138dbc9b955" path="/var/lib/kubelet/pods/980a30c0-fdfa-4d75-906e-c138dbc9b955/volumes" Oct 30 01:58:26.730569 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:26.730531 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:27.274043 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:27.274011 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:28.280279 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:28.280248 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp" Oct 30 01:58:29.015485 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:29.015446 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:30.376117 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:30.376072 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:30.858854 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:30.858818 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:30.974868 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:30.974837 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:31.143974 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:31.143881 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:33.017557 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:33.017519 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:33.437113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:33.428120 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:58:34.418362 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:34.418322 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:34.999528 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:34.999492 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:35.140019 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:35.139979 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:35.280106 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:35.280008 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:36.039836 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:36.039798 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb"] Oct 30 01:58:36.052006 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:36.051979 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24cf977b-sgfnb"] Oct 30 01:58:36.541287 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:36.541241 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85cfaa98-fbb1-46c9-800e-1a5a190abd15" path="/var/lib/kubelet/pods/85cfaa98-fbb1-46c9-800e-1a5a190abd15/volumes" Oct 30 01:58:36.829660 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:36.829567 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:36.940049 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:36.939670 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:37.074007 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:37.073973 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:37.859498 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:37.859465 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:38.015911 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:38.015861 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:38.068156 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:38.068125 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s"] Oct 30 01:58:38.083974 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:38.083934 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecbf2724-fbv6s"] Oct 30 01:58:38.568767 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:38.568727 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:38.593142 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:38.587718 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d1c517a-5c98-447e-b804-75933d6f64bf" path="/var/lib/kubelet/pods/8d1c517a-5c98-447e-b804-75933d6f64bf/volumes" Oct 30 01:58:40.436910 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:40.436870 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:42.092764 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:42.092726 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:43.110346 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:43.110314 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:43.325903 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:43.278108 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:43.416735 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:43.416656 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:44.110266 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:44.110232 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:45.717173 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:45.717136 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:46.022740 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:46.022658 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:46.558407 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:46.558374 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:46.750847 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:46.750811 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:47.383004 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:47.382970 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:47.829027 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:47.828977 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx"] Oct 30 01:58:47.829858 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:47.829824 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" podUID="fa5cf05f-5444-459e-ba19-657e8c9f4ee1" containerName="istio-proxy" containerID="cri-o://4c84ea911af364487e52602de6b1278ce8bd344706e4e06a4a5d9f0201414449" gracePeriod=30 Oct 30 01:58:47.850418 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:47.850386 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:48.617514 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:48.617480 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:48.763613 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:48.763563 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:49.261892 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:49.261844 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:49.631894 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:49.631786 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:51.175015 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:51.174980 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:52.035147 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:52.035071 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw"] Oct 30 01:58:52.046050 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:52.046018 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2de2e31-tgcqw"] Oct 30 01:58:52.543455 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:52.543414 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7239897c-d0c8-4367-8ec3-cb0320eab379" path="/var/lib/kubelet/pods/7239897c-d0c8-4367-8ec3-cb0320eab379/volumes" Oct 30 01:58:52.728073 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:52.728044 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:53.148627 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.148595 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:58:53.400581 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.400512 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:58:53.506985 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.506942 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-workload-certs\") pod \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " Oct 30 01:58:53.507204 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.507016 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-envoy\") pod \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " Oct 30 01:58:53.507204 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.507162 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-credential-socket\") pod \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " Oct 30 01:58:53.507334 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.507221 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-podinfo\") pod \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " Oct 30 01:58:53.507394 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.507355 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-workload-socket\") pod \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " Oct 30 01:58:53.507394 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.507385 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-token\" (UniqueName: \"kubernetes.io/projected/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-token\") pod \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " Oct 30 01:58:53.507498 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.507411 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-data\") pod \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " Oct 30 01:58:53.507498 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.507494 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istiod-ca-cert\") pod \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " Oct 30 01:58:53.507609 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.507599 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-kube-api-access-cs7tl\" (UniqueName: \"kubernetes.io/projected/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-kube-api-access-cs7tl\") pod \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\" (UID: \"fa5cf05f-5444-459e-ba19-657e8c9f4ee1\") " Oct 30 01:58:53.507950 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.507920 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-workload-certs" pod "fa5cf05f-5444-459e-ba19-657e8c9f4ee1" (UID: "fa5cf05f-5444-459e-ba19-657e8c9f4ee1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:53.508377 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.508343 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:58:53.508623 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.508587 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-credential-socket" pod "fa5cf05f-5444-459e-ba19-657e8c9f4ee1" (UID: "fa5cf05f-5444-459e-ba19-657e8c9f4ee1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:53.508814 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.508792 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-workload-socket" pod "fa5cf05f-5444-459e-ba19-657e8c9f4ee1" (UID: "fa5cf05f-5444-459e-ba19-657e8c9f4ee1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:53.508999 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.508977 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-data" pod "fa5cf05f-5444-459e-ba19-657e8c9f4ee1" (UID: "fa5cf05f-5444-459e-ba19-657e8c9f4ee1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:53.516027 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.509314 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istiod-ca-cert" pod "fa5cf05f-5444-459e-ba19-657e8c9f4ee1" (UID: "fa5cf05f-5444-459e-ba19-657e8c9f4ee1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:53.516027 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.510749 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-kube-api-access-cs7tl" pod "fa5cf05f-5444-459e-ba19-657e8c9f4ee1" (UID: "fa5cf05f-5444-459e-ba19-657e8c9f4ee1"). InnerVolumeSpecName "kube-api-access-cs7tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:53.523046 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.523008 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-envoy" pod "fa5cf05f-5444-459e-ba19-657e8c9f4ee1" (UID: "fa5cf05f-5444-459e-ba19-657e8c9f4ee1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:58:53.523217 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.523186 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-token" pod "fa5cf05f-5444-459e-ba19-657e8c9f4ee1" (UID: "fa5cf05f-5444-459e-ba19-657e8c9f4ee1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:53.523319 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.523283 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-podinfo" pod "fa5cf05f-5444-459e-ba19-657e8c9f4ee1" (UID: "fa5cf05f-5444-459e-ba19-657e8c9f4ee1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:58:53.609582 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.609549 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs7tl\" (UniqueName: \"kubernetes.io/projected/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-kube-api-access-cs7tl\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:58:53.609582 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.609576 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:58:53.609582 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.609589 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:58:53.610030 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.609600 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:58:53.610030 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.609609 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:58:53.610030 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.609617 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:58:53.610030 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.609624 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:58:53.610030 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.609632 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa5cf05f-5444-459e-ba19-657e8c9f4ee1-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:58:53.722391 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.721353 5998 generic.go:356] "Generic (PLEG): container finished" podID="fa5cf05f-5444-459e-ba19-657e8c9f4ee1" containerID="4c84ea911af364487e52602de6b1278ce8bd344706e4e06a4a5d9f0201414449" exitCode=0 Oct 30 01:58:53.722391 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.721599 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" event={"ID":"fa5cf05f-5444-459e-ba19-657e8c9f4ee1","Type":"ContainerDied","Data":"4c84ea911af364487e52602de6b1278ce8bd344706e4e06a4a5d9f0201414449"} Oct 30 01:58:53.722391 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.721653 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" event={"ID":"fa5cf05f-5444-459e-ba19-657e8c9f4ee1","Type":"ContainerDied","Data":"2edfa97e651c60aed441496742ae317490cf0730144108be3b9b6f4a54357371"} Oct 30 01:58:53.722391 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.721682 5998 scope.go:122] "RemoveContainer" containerID="4c84ea911af364487e52602de6b1278ce8bd344706e4e06a4a5d9f0201414449" Oct 30 01:58:53.722391 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.721851 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx" Oct 30 01:58:53.749274 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.749242 5998 scope.go:122] "RemoveContainer" containerID="e800b0c096f46733333b293bfa3c5d29fa9c7af0475ef21250822a59d77bd070" Oct 30 01:58:53.761552 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.761531 5998 scope.go:122] "RemoveContainer" containerID="4c84ea911af364487e52602de6b1278ce8bd344706e4e06a4a5d9f0201414449" Oct 30 01:58:53.761891 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:58:53.761872 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c84ea911af364487e52602de6b1278ce8bd344706e4e06a4a5d9f0201414449\": container with ID starting with 4c84ea911af364487e52602de6b1278ce8bd344706e4e06a4a5d9f0201414449 not found: ID does not exist" containerID="4c84ea911af364487e52602de6b1278ce8bd344706e4e06a4a5d9f0201414449" Oct 30 01:58:53.761956 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.761900 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c84ea911af364487e52602de6b1278ce8bd344706e4e06a4a5d9f0201414449"} err="failed to get container status \"4c84ea911af364487e52602de6b1278ce8bd344706e4e06a4a5d9f0201414449\": rpc error: code = NotFound desc = could not find container \"4c84ea911af364487e52602de6b1278ce8bd344706e4e06a4a5d9f0201414449\": container with ID starting with 4c84ea911af364487e52602de6b1278ce8bd344706e4e06a4a5d9f0201414449 not found: ID does not exist" Oct 30 01:58:53.761956 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.761918 5998 scope.go:122] "RemoveContainer" containerID="e800b0c096f46733333b293bfa3c5d29fa9c7af0475ef21250822a59d77bd070" Oct 30 01:58:53.762263 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:58:53.762242 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e800b0c096f46733333b293bfa3c5d29fa9c7af0475ef21250822a59d77bd070\": container with ID starting with e800b0c096f46733333b293bfa3c5d29fa9c7af0475ef21250822a59d77bd070 not found: ID does not exist" containerID="e800b0c096f46733333b293bfa3c5d29fa9c7af0475ef21250822a59d77bd070" Oct 30 01:58:53.762331 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.762274 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e800b0c096f46733333b293bfa3c5d29fa9c7af0475ef21250822a59d77bd070"} err="failed to get container status \"e800b0c096f46733333b293bfa3c5d29fa9c7af0475ef21250822a59d77bd070\": rpc error: code = NotFound desc = could not find container \"e800b0c096f46733333b293bfa3c5d29fa9c7af0475ef21250822a59d77bd070\": container with ID starting with e800b0c096f46733333b293bfa3c5d29fa9c7af0475ef21250822a59d77bd070 not found: ID does not exist" Oct 30 01:58:53.771645 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.771613 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:53.778503 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.778479 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx"] Oct 30 01:58:53.790183 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.790157 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2818a6-openshift-default-bf45b947f-kbtxx"] Oct 30 01:58:53.929113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:53.920763 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:54.070102 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:54.070011 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:54.539355 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:54.539323 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa5cf05f-5444-459e-ba19-657e8c9f4ee1" path="/var/lib/kubelet/pods/fa5cf05f-5444-459e-ba19-657e8c9f4ee1/volumes" Oct 30 01:58:55.037235 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:55.037198 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz"] Oct 30 01:58:55.047384 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:55.047349 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f869d79-tgbxz"] Oct 30 01:58:55.277234 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:58:55.277190 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:58:55.914651 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:55.914617 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:56.070980 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:56.070947 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:56.185655 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:56.185550 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:56.541619 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:56.541537 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2" path="/var/lib/kubelet/pods/69fea0c4-aa7f-47e8-b1d8-ceaf031a20a2/volumes" Oct 30 01:58:57.976050 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:57.976017 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:59.131071 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.131037 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:59.296186 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.296013 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:58:59.711528 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.711474 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w"] Oct 30 01:58:59.712459 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.712436 5998 cpu_manager.go:414] "containerMap: removing container" podUID="fa5cf05f-5444-459e-ba19-657e8c9f4ee1" containerName="istio-proxy" Oct 30 01:58:59.712572 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.712462 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa5cf05f-5444-459e-ba19-657e8c9f4ee1" containerName="istio-proxy" Oct 30 01:58:59.712630 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.712570 5998 cpu_manager.go:414] "containerMap: removing container" podUID="fa5cf05f-5444-459e-ba19-657e8c9f4ee1" containerName="istio-proxy" Oct 30 01:58:59.712630 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.712584 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa5cf05f-5444-459e-ba19-657e8c9f4ee1" containerName="istio-proxy" Oct 30 01:58:59.712772 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.712758 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa5cf05f-5444-459e-ba19-657e8c9f4ee1" containerName="istio-proxy" Oct 30 01:58:59.712823 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.712780 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa5cf05f-5444-459e-ba19-657e8c9f4ee1" containerName="istio-proxy" Oct 30 01:58:59.721241 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.721192 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" Oct 30 01:58:59.742880 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.739570 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w"] Oct 30 01:58:59.742880 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.740021 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7542fa\"" Oct 30 01:58:59.756428 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.756389 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7542fa-dockercfg-bhh9x\"" Oct 30 01:58:59.858774 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.858738 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f80e088-78c9-4b95-98cc-4d1dd67d9a34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w\" (UID: \"4f80e088-78c9-4b95-98cc-4d1dd67d9a34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" Oct 30 01:58:59.858954 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.858801 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97clm\" (UniqueName: \"kubernetes.io/projected/4f80e088-78c9-4b95-98cc-4d1dd67d9a34-kube-api-access-97clm\") pod \"payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w\" (UID: \"4f80e088-78c9-4b95-98cc-4d1dd67d9a34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" Oct 30 01:58:59.959986 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.959953 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f80e088-78c9-4b95-98cc-4d1dd67d9a34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w\" (UID: \"4f80e088-78c9-4b95-98cc-4d1dd67d9a34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" Oct 30 01:58:59.960204 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.960021 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97clm\" (UniqueName: \"kubernetes.io/projected/4f80e088-78c9-4b95-98cc-4d1dd67d9a34-kube-api-access-97clm\") pod \"payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w\" (UID: \"4f80e088-78c9-4b95-98cc-4d1dd67d9a34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" Oct 30 01:58:59.960737 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.960718 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f80e088-78c9-4b95-98cc-4d1dd67d9a34-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w\" (UID: \"4f80e088-78c9-4b95-98cc-4d1dd67d9a34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" Oct 30 01:58:59.990395 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:58:59.990322 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97clm\" (UniqueName: \"kubernetes.io/projected/4f80e088-78c9-4b95-98cc-4d1dd67d9a34-kube-api-access-97clm\") pod \"payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w\" (UID: \"4f80e088-78c9-4b95-98cc-4d1dd67d9a34\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" Oct 30 01:59:00.060471 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:00.060438 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" Oct 30 01:59:00.394444 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:00.394414 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w"] Oct 30 01:59:00.396605 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:59:00.396580 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f80e088_78c9_4b95_98cc_4d1dd67d9a34.slice/crio-02a8282d351060c0d014ad71d12824317a0d6e1c5cb9529698e8fe7893ee9ced WatchSource:0}: Error finding container 02a8282d351060c0d014ad71d12824317a0d6e1c5cb9529698e8fe7893ee9ced: Status 404 returned error can't find the container with id 02a8282d351060c0d014ad71d12824317a0d6e1c5cb9529698e8fe7893ee9ced Oct 30 01:59:00.427178 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:00.427152 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:00.783083 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:00.783029 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" event={"ID":"4f80e088-78c9-4b95-98cc-4d1dd67d9a34","Type":"ContainerStarted","Data":"d7e83251aa9f12aefcf1bceefb3fb3461812edcb31fdffb834c6f1bcd29d7188"} Oct 30 01:59:00.783287 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:00.783124 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" event={"ID":"4f80e088-78c9-4b95-98cc-4d1dd67d9a34","Type":"ContainerStarted","Data":"02a8282d351060c0d014ad71d12824317a0d6e1c5cb9529698e8fe7893ee9ced"} Oct 30 01:59:00.783287 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:00.783162 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" Oct 30 01:59:00.828977 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:00.828912 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" podStartSLOduration=1.82889034 podStartE2EDuration="1.82889034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:00.822812861 +0000 UTC m=+1770.401603943" watchObservedRunningTime="2026-10-30 01:59:00.82889034 +0000 UTC m=+1770.407681424" Oct 30 01:59:02.102955 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:02.102913 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:02.232180 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:02.223665 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:02.804170 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:02.804131 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:03.179150 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:03.177540 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:03.946937 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:03.946900 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:04.048550 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.048504 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn"] Oct 30 01:59:04.059213 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.059181 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22c47e8f-pv9wn"] Oct 30 01:59:04.160027 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.159992 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:04.511133 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.494688 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76"] Oct 30 01:59:04.517265 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.517230 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.539279 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.538260 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b273b6-openshift-default-dockercfg-76lwh\"" Oct 30 01:59:04.539279 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.538963 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76"] Oct 30 01:59:04.632503 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.629918 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-credential-socket\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.632503 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.629993 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-podinfo\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.632503 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.630051 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrblp\" (UniqueName: \"kubernetes.io/projected/323ecfe9-7182-4e29-9774-dd3dd728fc09-kube-api-access-rrblp\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.632503 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.630228 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-token\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.632503 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.630261 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-data\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.632503 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.630294 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-envoy\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.632503 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.630336 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-workload-certs\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.632503 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.630389 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-workload-socket\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.632503 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.630497 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323ecfe9-7182-4e29-9774-dd3dd728fc09-istiod-ca-cert\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.652525 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.645922 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:04.675236 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.675179 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f51353d2-1f84-494e-829f-1d7eb6ed14d1" path="/var/lib/kubelet/pods/f51353d2-1f84-494e-829f-1d7eb6ed14d1/volumes" Oct 30 01:59:04.731795 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.731755 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-token\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.733023 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.732986 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-data\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.733177 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.733035 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-envoy\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.733177 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.733074 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-workload-certs\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.733177 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.733142 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-workload-socket\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.733359 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.733246 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323ecfe9-7182-4e29-9774-dd3dd728fc09-istiod-ca-cert\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.733359 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.733320 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-credential-socket\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.733464 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.733373 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-podinfo\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.733464 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.733428 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrblp\" (UniqueName: \"kubernetes.io/projected/323ecfe9-7182-4e29-9774-dd3dd728fc09-kube-api-access-rrblp\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.734142 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.734118 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-data\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.734328 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.734305 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-workload-socket\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.734951 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.734927 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323ecfe9-7182-4e29-9774-dd3dd728fc09-istiod-ca-cert\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.735285 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.735205 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-credential-socket\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.735877 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.735853 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-workload-certs\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.742554 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.742518 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-envoy\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.761279 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.761197 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-podinfo\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.762472 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.762432 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:04.764949 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.764889 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-token\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.769577 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.769498 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrblp\" (UniqueName: \"kubernetes.io/projected/323ecfe9-7182-4e29-9774-dd3dd728fc09-kube-api-access-rrblp\") pod \"e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:04.863636 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:04.863601 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:05.196400 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:05.196315 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76"] Oct 30 01:59:05.200476 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:59:05.200440 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323ecfe9_7182_4e29_9774_dd3dd728fc09.slice/crio-fe3e1d9aa330adef2317a80ce920358909c7b4e10aa880d2a3587b93ac307be5 WatchSource:0}: Error finding container fe3e1d9aa330adef2317a80ce920358909c7b4e10aa880d2a3587b93ac307be5: Status 404 returned error can't find the container with id fe3e1d9aa330adef2317a80ce920358909c7b4e10aa880d2a3587b93ac307be5 Oct 30 01:59:05.204266 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:05.204221 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:59:05.204395 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:05.204345 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:59:05.204395 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:05.204388 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:59:05.778748 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:05.778715 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:05.858112 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:05.857955 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" event={"ID":"323ecfe9-7182-4e29-9774-dd3dd728fc09","Type":"ContainerStarted","Data":"aa45e4b085d2175b53a045d6483a965484aea40259e90f4dabb71fea592559e1"} Oct 30 01:59:05.858112 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:05.858023 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" event={"ID":"323ecfe9-7182-4e29-9774-dd3dd728fc09","Type":"ContainerStarted","Data":"fe3e1d9aa330adef2317a80ce920358909c7b4e10aa880d2a3587b93ac307be5"} Oct 30 01:59:05.863741 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:05.863713 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:05.920416 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:05.920348 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" podStartSLOduration=1.920323363 podStartE2EDuration="1.920323363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:05.915236596 +0000 UTC m=+1775.494027678" watchObservedRunningTime="2026-10-30 01:59:05.920323363 +0000 UTC m=+1775.499114445" Oct 30 01:59:06.869736 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:06.869700 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:07.825305 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:07.825262 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:07.884376 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:07.884346 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:07.892215 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:07.892182 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 01:59:08.567854 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:08.567813 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:08.689413 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:08.689352 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:10.454819 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:10.454787 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:10.617197 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:10.617123 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:11.793327 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:11.793296 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" Oct 30 01:59:12.542663 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:12.542635 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:12.653399 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:12.653365 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:12.766279 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:12.766244 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:14.336486 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:14.336443 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:15.091182 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:15.091124 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:15.584287 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:15.584235 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:15.734604 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:15.734568 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:15.881728 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:15.881636 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:15.998929 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:15.998890 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67"] Oct 30 01:59:16.008121 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.007710 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.030070 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.029469 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67"] Oct 30 01:59:16.036284 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.036252 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-86dd79-openshift-default-dockercfg-4zq9k\"" Oct 30 01:59:16.146009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.145898 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-data\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.146009 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.145969 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-credential-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.146305 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.146130 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkqgq\" (UniqueName: \"kubernetes.io/projected/46590fa3-d39f-4df5-8c5b-96520faa7625-kube-api-access-gkqgq\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.146305 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.146198 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-token\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.146305 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.146271 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-workload-certs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.146477 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.146346 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46590fa3-d39f-4df5-8c5b-96520faa7625-istiod-ca-cert\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.146477 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.146378 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-podinfo\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.146477 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.146427 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-workload-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.146620 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.146502 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-envoy\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.248019 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.247983 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-envoy\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.248270 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.248068 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-data\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.248270 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.248121 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-credential-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.248270 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.248159 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkqgq\" (UniqueName: \"kubernetes.io/projected/46590fa3-d39f-4df5-8c5b-96520faa7625-kube-api-access-gkqgq\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.248270 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.248190 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-token\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.248270 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.248237 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-workload-certs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.248557 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.248298 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46590fa3-d39f-4df5-8c5b-96520faa7625-istiod-ca-cert\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.248557 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.248318 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-podinfo\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.248557 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.248348 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-workload-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.248815 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.248779 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-workload-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.249793 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.249332 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-workload-certs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.249793 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.249663 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-data\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.249973 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.249901 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-credential-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.250041 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.249970 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46590fa3-d39f-4df5-8c5b-96520faa7625-istiod-ca-cert\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.252304 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.252246 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-envoy\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.254061 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.254016 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-podinfo\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.283383 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.283312 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-token\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.300146 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.286176 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkqgq\" (UniqueName: \"kubernetes.io/projected/46590fa3-d39f-4df5-8c5b-96520faa7625-kube-api-access-gkqgq\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.311008 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.310972 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf"] Oct 30 01:59:16.332141 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.324039 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.332850 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.332767 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf"] Oct 30 01:59:16.349127 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.347600 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:16.365146 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.363808 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5c90f5-openshift-default-dockercfg-ql2vd\"" Oct 30 01:59:16.411236 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.411202 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:16.453932 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.453113 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-token\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.453932 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.453216 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-workload-certs\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.453932 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.453258 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-credential-socket\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.453932 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.453282 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-data\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.453932 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.453428 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hv6q\" (UniqueName: \"kubernetes.io/projected/f8e03b95-821f-456c-a435-831d0bfa52f0-kube-api-access-4hv6q\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.453932 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.453473 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-workload-socket\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.453932 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.453530 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-podinfo\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.453932 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.453596 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-envoy\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.453932 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.453664 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8e03b95-821f-456c-a435-831d0bfa52f0-istiod-ca-cert\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.552757 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.557142 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-workload-certs\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.557199 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-credential-socket\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.557226 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-data\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.557379 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hv6q\" (UniqueName: \"kubernetes.io/projected/f8e03b95-821f-456c-a435-831d0bfa52f0-kube-api-access-4hv6q\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.557424 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-workload-socket\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.557484 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-podinfo\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.557553 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-envoy\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.557606 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8e03b95-821f-456c-a435-831d0bfa52f0-istiod-ca-cert\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.557688 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-token\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.557744 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-credential-socket\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.558037 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-data\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.558118 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-workload-socket\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.558775 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8e03b95-821f-456c-a435-831d0bfa52f0-istiod-ca-cert\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.559812 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-workload-certs\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.561766 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-envoy\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.564416 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-podinfo\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.590073 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hv6q\" (UniqueName: \"kubernetes.io/projected/f8e03b95-821f-456c-a435-831d0bfa52f0-kube-api-access-4hv6q\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.606884 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.593458 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-token\") pod \"e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.674573 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.674492 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:16.675594 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.675226 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:16.812768 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.812467 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67"] Oct 30 01:59:16.845432 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:59:16.845362 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46590fa3_d39f_4df5_8c5b_96520faa7625.slice/crio-26b9f0d4e80b9f15e6b291722d6391c899fb1fb37c6a91f16e79cfbecc44ecac WatchSource:0}: Error finding container 26b9f0d4e80b9f15e6b291722d6391c899fb1fb37c6a91f16e79cfbecc44ecac: Status 404 returned error can't find the container with id 26b9f0d4e80b9f15e6b291722d6391c899fb1fb37c6a91f16e79cfbecc44ecac Oct 30 01:59:16.848707 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.848650 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:59:16.848852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.848751 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:59:16.848852 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.848795 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:59:16.991169 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.991128 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" event={"ID":"46590fa3-d39f-4df5-8c5b-96520faa7625","Type":"ContainerStarted","Data":"c0ccbf7540ec62ffa995f28751b050f8eacaef1531ccca6f18ffbbdc1eda561a"} Oct 30 01:59:16.991319 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:16.991194 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" event={"ID":"46590fa3-d39f-4df5-8c5b-96520faa7625","Type":"ContainerStarted","Data":"26b9f0d4e80b9f15e6b291722d6391c899fb1fb37c6a91f16e79cfbecc44ecac"} Oct 30 01:59:17.026469 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:17.026049 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf"] Oct 30 01:59:17.029748 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:59:17.029699 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e03b95_821f_456c_a435_831d0bfa52f0.slice/crio-2e7458336187837c34141481eeaa049b0ea014c570cbaf09b21b7a0818cf30e9 WatchSource:0}: Error finding container 2e7458336187837c34141481eeaa049b0ea014c570cbaf09b21b7a0818cf30e9: Status 404 returned error can't find the container with id 2e7458336187837c34141481eeaa049b0ea014c570cbaf09b21b7a0818cf30e9 Oct 30 01:59:17.034077 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:17.033486 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:59:17.034077 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:17.033571 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:59:17.034077 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:17.033615 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:59:17.048571 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:17.048515 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" podStartSLOduration=2.048496007 podStartE2EDuration="2.048496007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:17.046172114 +0000 UTC m=+1786.624963196" watchObservedRunningTime="2026-10-30 01:59:17.048496007 +0000 UTC m=+1786.627287091" Oct 30 01:59:17.349885 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:17.349804 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:17.351633 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:17.351600 5998 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 01:59:17.351821 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:17.351658 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" podUID="46590fa3-d39f-4df5-8c5b-96520faa7625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 01:59:18.011114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:18.009076 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" event={"ID":"f8e03b95-821f-456c-a435-831d0bfa52f0","Type":"ContainerStarted","Data":"9ca51b1842e73bbb0337773e16edb2615575f589a2c25aaaeaaa959531c4aa23"} Oct 30 01:59:18.011114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:18.009162 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" event={"ID":"f8e03b95-821f-456c-a435-831d0bfa52f0","Type":"ContainerStarted","Data":"2e7458336187837c34141481eeaa049b0ea014c570cbaf09b21b7a0818cf30e9"} Oct 30 01:59:18.068938 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:18.068879 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" podStartSLOduration=2.06886461 podStartE2EDuration="2.06886461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:18.065335848 +0000 UTC m=+1787.644126931" watchObservedRunningTime="2026-10-30 01:59:18.06886461 +0000 UTC m=+1787.647655690" Oct 30 01:59:18.354188 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:18.352686 5998 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 01:59:18.354188 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:18.352746 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" podUID="46590fa3-d39f-4df5-8c5b-96520faa7625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 01:59:18.505607 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:18.505545 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:18.676294 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:18.676260 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:18.683109 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:18.682006 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:19.035183 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:19.026871 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:19.035594 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:19.035368 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 01:59:19.350350 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:19.350247 5998 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 01:59:19.350350 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:19.350327 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" podUID="46590fa3-d39f-4df5-8c5b-96520faa7625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 01:59:20.350390 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:20.350343 5998 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 01:59:20.350919 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:20.350411 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" podUID="46590fa3-d39f-4df5-8c5b-96520faa7625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 01:59:20.543949 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:20.543907 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:20.677118 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:20.677053 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:20.828113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:20.820218 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:20.950615 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:20.947978 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:21.084113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:21.072612 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:21.366320 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:21.350302 5998 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 01:59:21.366320 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:21.350372 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" podUID="46590fa3-d39f-4df5-8c5b-96520faa7625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 01:59:21.748119 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:21.741984 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:21.975041 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:21.975008 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz"] Oct 30 01:59:21.979303 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:21.979276 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:21.994591 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:21.994563 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 01:59:22.028646 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.023427 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-v6t8d\"" Oct 30 01:59:22.028646 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.023481 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 01:59:22.028646 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.023511 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 01:59:22.063672 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.063617 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.064058 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.064034 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.064134 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.064073 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv6zk\" (UniqueName: \"kubernetes.io/projected/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-kube-api-access-nv6zk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.064300 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.064226 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.065279 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.065253 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.065351 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.065314 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.065426 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.065411 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.069673 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.069649 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz"] Oct 30 01:59:22.167110 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.167057 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.167324 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.167200 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.167324 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.167219 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6zk\" (UniqueName: \"kubernetes.io/projected/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-kube-api-access-nv6zk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.167324 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.167264 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.167494 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.167326 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.167494 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.167354 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.167494 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.167418 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.168456 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.168429 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.169604 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.169580 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.170583 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.170551 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.170939 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.170919 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.171024 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.171007 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.199544 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.199515 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv6zk\" (UniqueName: \"kubernetes.io/projected/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-kube-api-access-nv6zk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.199544 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.199535 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bt6gz\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.293677 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.293583 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:22.350681 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.350423 5998 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 01:59:22.350681 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.350499 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" podUID="46590fa3-d39f-4df5-8c5b-96520faa7625" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 01:59:22.651198 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.651169 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz"] Oct 30 01:59:22.653496 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:59:22.653458 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bfb5bd1_c7a6_4760_b8b9_c26e06560652.slice/crio-8c038d10d52ca7a847ceb36d1ff4853d325c50854ed789e0901ff901b2a81df2 WatchSource:0}: Error finding container 8c038d10d52ca7a847ceb36d1ff4853d325c50854ed789e0901ff901b2a81df2: Status 404 returned error can't find the container with id 8c038d10d52ca7a847ceb36d1ff4853d325c50854ed789e0901ff901b2a81df2 Oct 30 01:59:22.824114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:22.823547 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:23.198110 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:23.197662 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" event={"ID":"1bfb5bd1-c7a6-4760-b8b9-c26e06560652","Type":"ContainerStarted","Data":"8c038d10d52ca7a847ceb36d1ff4853d325c50854ed789e0901ff901b2a81df2"} Oct 30 01:59:23.591353 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:23.591247 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:23.618113 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:23.618069 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:23.619003 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:23.618982 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:23.620134 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:23.620078 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 01:59:24.069434 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:24.069401 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:25.206316 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:25.206273 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:25.227750 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:59:25.227711 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:59:25.388412 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:25.388374 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:26.170620 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:26.170588 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:27.883227 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:27.883191 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:28.034117 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:28.034067 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62"] Oct 30 01:59:28.043610 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:28.043580 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48b7ef7b-l5x62"] Oct 30 01:59:28.542863 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:28.542827 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1" path="/var/lib/kubelet/pods/f721b93a-2a8e-4aaf-8e6a-6df0318d4fd1/volumes" Oct 30 01:59:28.955799 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:28.955750 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 01:59:29.762756 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:29.760674 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:29.878535 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:29.878493 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:29.997523 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:29.997491 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:31.296810 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:31.296716 5998 scope.go:122] "RemoveContainer" containerID="bddde80c16b650abd930a4e97430f726f4ccd5c274455feda7514bb632ceed9f" Oct 30 01:59:31.311339 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:31.311310 5998 scope.go:122] "RemoveContainer" containerID="6d2d7914acc6013698a609befc657f2842d149174916bcd8aa32051eb19d6bbe" Oct 30 01:59:31.328505 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:31.328483 5998 scope.go:122] "RemoveContainer" containerID="bee4f4a5610b9da77279c70f7836ca4c771a0e575e0a7848886b66e64c7334df" Oct 30 01:59:31.339550 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:31.339523 5998 scope.go:122] "RemoveContainer" containerID="c8be907f199dcb80db3186b1dbcb0c712aff44dab624456a132eeae19bd35717" Oct 30 01:59:31.352231 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:31.352188 5998 scope.go:122] "RemoveContainer" containerID="c7d7783dfe04e555140bc714a3053fe840e9b5757db5aeeb3061082b695e96cf" Oct 30 01:59:31.367030 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:31.366869 5998 scope.go:122] "RemoveContainer" containerID="8d8bb76c08e7ca811b25dc3cd8ba6ceac8b23643991a296d856623fe6dcc0888" Oct 30 01:59:31.390065 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:31.389920 5998 scope.go:122] "RemoveContainer" containerID="f1a6b8fe5b5061ccd65c60e781c0272ba160b4db1ee18bed3e0b22c4b1e1042a" Oct 30 01:59:31.810644 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:31.810607 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:33.169390 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:33.169346 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:59:33.169731 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:33.169427 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:59:33.344430 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:33.344393 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" event={"ID":"1bfb5bd1-c7a6-4760-b8b9-c26e06560652","Type":"ContainerStarted","Data":"05b9eb9b80e37f987115bd3a54424b6abecd9f024a51f800160256d8725b3ec8"} Oct 30 01:59:33.344612 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:33.344465 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:33.414070 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:33.414017 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" podStartSLOduration=1.906921675 podStartE2EDuration="12.414003458s" totalImagesPullingTime="10.507081783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:33.401571639 +0000 UTC m=+1802.980362721" watchObservedRunningTime="2026-10-30 01:59:33.414003458 +0000 UTC m=+1802.992794535" Oct 30 01:59:34.151146 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:34.149284 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:34.356498 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:34.356463 5998 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bt6gz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:59:34.356995 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:34.356518 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" podUID="1bfb5bd1-c7a6-4760-b8b9-c26e06560652" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:59:35.226244 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:35.226205 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:35.718182 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:35.718147 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:37.356473 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:37.356430 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 01:59:37.710861 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:37.710823 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:39.120701 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:39.120663 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:39.242443 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:39.242405 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:39.351236 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:39.351207 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:39.459266 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:39.459235 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:40.465590 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:40.465553 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:42.883267 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:42.883171 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:45.143400 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:45.143357 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:45.261941 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:45.261893 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:45.533725 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:45.533688 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:46.117220 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.117182 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc"] Oct 30 01:59:46.122161 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.122128 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.150017 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.149790 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc"] Oct 30 01:59:46.268186 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.268143 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-data\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.268361 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.268216 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-credential-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.268477 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.268454 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-token\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.268578 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.268559 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-podinfo\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.268629 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.268618 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svj7g\" (UniqueName: \"kubernetes.io/projected/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-kube-api-access-svj7g\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.268744 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.268724 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istiod-ca-cert\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.268904 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.268882 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-workload-certs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.269010 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.268997 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-envoy\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.269060 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.269048 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-workload-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.374702 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.374603 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-token\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.375566 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.375537 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-podinfo\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.381082 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.381052 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svj7g\" (UniqueName: \"kubernetes.io/projected/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-kube-api-access-svj7g\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.382209 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.381706 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istiod-ca-cert\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.382604 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.382577 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-workload-certs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.383480 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.383459 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-envoy\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.384074 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.383201 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istiod-ca-cert\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.384204 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.383212 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-workload-certs\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.384276 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.380073 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-podinfo\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.384276 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.384263 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-workload-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.384399 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.384344 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-data\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.384702 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.384681 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-data\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.384899 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.384881 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-workload-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.384996 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.384966 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-credential-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.385346 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.385325 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-credential-socket\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.386764 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.386744 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-envoy\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.398032 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.398007 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-token\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.407465 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.407434 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svj7g\" (UniqueName: \"kubernetes.io/projected/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-kube-api-access-svj7g\") pod \"e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.442595 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.442559 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:46.786938 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.786909 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc"] Oct 30 01:59:46.789308 ip-10-0-130-0 kubenswrapper[5998]: W1030 01:59:46.789279 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bebf03_9dfe_4acf_a5e7_b4e8e6ec65f8.slice/crio-f1cf8b27f5dbe4fd8abec51fad0ad2b375bdef621cccf80f1450fe9de07a26d0 WatchSource:0}: Error finding container f1cf8b27f5dbe4fd8abec51fad0ad2b375bdef621cccf80f1450fe9de07a26d0: Status 404 returned error can't find the container with id f1cf8b27f5dbe4fd8abec51fad0ad2b375bdef621cccf80f1450fe9de07a26d0 Oct 30 01:59:46.791969 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.791939 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:59:46.792061 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.792005 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:59:46.792061 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:46.792032 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 01:59:47.004756 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:47.004724 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:47.174157 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:47.174120 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:47.488543 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:47.488445 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" event={"ID":"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8","Type":"ContainerStarted","Data":"540f538545780d1b5e7253eddbc9122328154546448d0e3393ecfe5ad14b82e2"} Oct 30 01:59:47.488543 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:47.488515 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" event={"ID":"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8","Type":"ContainerStarted","Data":"f1cf8b27f5dbe4fd8abec51fad0ad2b375bdef621cccf80f1450fe9de07a26d0"} Oct 30 01:59:47.565932 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:47.565859 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" podStartSLOduration=1.5658343430000001 podStartE2EDuration="1.565834343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:47.548908156 +0000 UTC m=+1817.127699238" watchObservedRunningTime="2026-10-30 01:59:47.565834343 +0000 UTC m=+1817.144625425" Oct 30 01:59:48.446765 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:48.446731 5998 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 01:59:48.453237 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:48.453177 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" podUID="19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 01:59:48.453808 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:48.453789 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:48.656776 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:48.656169 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw"] Oct 30 01:59:48.656776 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:48.656491 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" podUID="925ff5f4-100e-46a4-a906-3a9ecef96765" containerName="istio-proxy" containerID="cri-o://3a91eb3f07c5ae57906745fbe42b211e39a26aee6823da7d6ef50d9355bdf9f7" gracePeriod=30 Oct 30 01:59:49.117114 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:49.116595 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:49.443992 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:49.443951 5998 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 01:59:49.444215 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:49.444022 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" podUID="19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 01:59:50.443650 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:50.443612 5998 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 01:59:50.444072 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:50.443696 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" podUID="19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 01:59:50.783245 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:50.783159 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:51.443896 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:51.443853 5998 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 01:59:51.444369 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:51.443918 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" podUID="19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 01:59:52.443711 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:52.443673 5998 patch_prober.go:29] interesting pod/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 01:59:52.443913 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:52.443729 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" podUID="19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 01:59:52.513783 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:52.513669 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:53.449198 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.449166 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:53.449677 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.449651 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:53.450308 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.450289 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 01:59:53.936838 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.936806 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:59:53.979268 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.979188 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-token\" (UniqueName: \"kubernetes.io/projected/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-token\") pod \"925ff5f4-100e-46a4-a906-3a9ecef96765\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " Oct 30 01:59:53.979268 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.979228 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/925ff5f4-100e-46a4-a906-3a9ecef96765-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925ff5f4-100e-46a4-a906-3a9ecef96765-istiod-ca-cert\") pod \"925ff5f4-100e-46a4-a906-3a9ecef96765\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " Oct 30 01:59:53.979503 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.979353 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-workload-socket\") pod \"925ff5f4-100e-46a4-a906-3a9ecef96765\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " Oct 30 01:59:53.979503 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.979476 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-podinfo\") pod \"925ff5f4-100e-46a4-a906-3a9ecef96765\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " Oct 30 01:59:53.979625 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.979540 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-credential-socket\") pod \"925ff5f4-100e-46a4-a906-3a9ecef96765\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " Oct 30 01:59:53.979625 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.979596 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/925ff5f4-100e-46a4-a906-3a9ecef96765-kube-api-access-q97zk\" (UniqueName: \"kubernetes.io/projected/925ff5f4-100e-46a4-a906-3a9ecef96765-kube-api-access-q97zk\") pod \"925ff5f4-100e-46a4-a906-3a9ecef96765\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " Oct 30 01:59:53.979731 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.979637 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-envoy\") pod \"925ff5f4-100e-46a4-a906-3a9ecef96765\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " Oct 30 01:59:53.979731 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.979692 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-workload-certs\") pod \"925ff5f4-100e-46a4-a906-3a9ecef96765\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " Oct 30 01:59:53.979843 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.979745 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-data\") pod \"925ff5f4-100e-46a4-a906-3a9ecef96765\" (UID: \"925ff5f4-100e-46a4-a906-3a9ecef96765\") " Oct 30 01:59:53.979950 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.979610 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-workload-socket" pod "925ff5f4-100e-46a4-a906-3a9ecef96765" (UID: "925ff5f4-100e-46a4-a906-3a9ecef96765"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:53.980070 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.979663 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925ff5f4-100e-46a4-a906-3a9ecef96765-istiod-ca-cert" pod "925ff5f4-100e-46a4-a906-3a9ecef96765" (UID: "925ff5f4-100e-46a4-a906-3a9ecef96765"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:53.980070 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.979894 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-credential-socket" pod "925ff5f4-100e-46a4-a906-3a9ecef96765" (UID: "925ff5f4-100e-46a4-a906-3a9ecef96765"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:53.980233 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.980183 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-workload-certs" pod "925ff5f4-100e-46a4-a906-3a9ecef96765" (UID: "925ff5f4-100e-46a4-a906-3a9ecef96765"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:53.980233 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.980205 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-data" pod "925ff5f4-100e-46a4-a906-3a9ecef96765" (UID: "925ff5f4-100e-46a4-a906-3a9ecef96765"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:53.980589 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.980564 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:59:53.980684 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.980593 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:59:53.980684 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.980609 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:59:53.980684 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.980623 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925ff5f4-100e-46a4-a906-3a9ecef96765-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:59:53.980684 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.980637 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:59:53.982059 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.982031 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-podinfo" pod "925ff5f4-100e-46a4-a906-3a9ecef96765" (UID: "925ff5f4-100e-46a4-a906-3a9ecef96765"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:59:53.982238 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.982054 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-token" pod "925ff5f4-100e-46a4-a906-3a9ecef96765" (UID: "925ff5f4-100e-46a4-a906-3a9ecef96765"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:53.982238 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.982112 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925ff5f4-100e-46a4-a906-3a9ecef96765-kube-api-access-q97zk" pod "925ff5f4-100e-46a4-a906-3a9ecef96765" (UID: "925ff5f4-100e-46a4-a906-3a9ecef96765"). InnerVolumeSpecName "kube-api-access-q97zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:53.982238 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:53.982084 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-envoy" pod "925ff5f4-100e-46a4-a906-3a9ecef96765" (UID: "925ff5f4-100e-46a4-a906-3a9ecef96765"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:59:54.081713 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:54.081672 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.081713 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:54.081708 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.081713 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:54.081717 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q97zk\" (UniqueName: \"kubernetes.io/projected/925ff5f4-100e-46a4-a906-3a9ecef96765-kube-api-access-q97zk\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.081957 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:54.081729 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925ff5f4-100e-46a4-a906-3a9ecef96765-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 01:59:54.296649 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:54.296561 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:54.586472 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:54.586432 5998 generic.go:356] "Generic (PLEG): container finished" podID="925ff5f4-100e-46a4-a906-3a9ecef96765" containerID="3a91eb3f07c5ae57906745fbe42b211e39a26aee6823da7d6ef50d9355bdf9f7" exitCode=0 Oct 30 01:59:54.586667 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:54.586502 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" Oct 30 01:59:54.586667 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:54.586563 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" event={"ID":"925ff5f4-100e-46a4-a906-3a9ecef96765","Type":"ContainerDied","Data":"3a91eb3f07c5ae57906745fbe42b211e39a26aee6823da7d6ef50d9355bdf9f7"} Oct 30 01:59:54.586667 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:54.586609 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw" event={"ID":"925ff5f4-100e-46a4-a906-3a9ecef96765","Type":"ContainerDied","Data":"045467e453ca173937713fcc7250644aebf2b99f6c897b3d78b593c7e6c2cbfd"} Oct 30 01:59:54.586667 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:54.586639 5998 scope.go:122] "RemoveContainer" containerID="3a91eb3f07c5ae57906745fbe42b211e39a26aee6823da7d6ef50d9355bdf9f7" Oct 30 01:59:54.603991 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:54.603965 5998 scope.go:122] "RemoveContainer" containerID="3a91eb3f07c5ae57906745fbe42b211e39a26aee6823da7d6ef50d9355bdf9f7" Oct 30 01:59:54.604378 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:59:54.604339 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a91eb3f07c5ae57906745fbe42b211e39a26aee6823da7d6ef50d9355bdf9f7\": container with ID starting with 3a91eb3f07c5ae57906745fbe42b211e39a26aee6823da7d6ef50d9355bdf9f7 not found: ID does not exist" containerID="3a91eb3f07c5ae57906745fbe42b211e39a26aee6823da7d6ef50d9355bdf9f7" Oct 30 01:59:54.604499 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:54.604384 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a91eb3f07c5ae57906745fbe42b211e39a26aee6823da7d6ef50d9355bdf9f7"} err="failed to get container status \"3a91eb3f07c5ae57906745fbe42b211e39a26aee6823da7d6ef50d9355bdf9f7\": rpc error: code = NotFound desc = could not find container \"3a91eb3f07c5ae57906745fbe42b211e39a26aee6823da7d6ef50d9355bdf9f7\": container with ID starting with 3a91eb3f07c5ae57906745fbe42b211e39a26aee6823da7d6ef50d9355bdf9f7 not found: ID does not exist" Oct 30 01:59:54.639432 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:54.639399 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw"] Oct 30 01:59:54.652292 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:54.652259 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-flwpw"] Oct 30 01:59:55.218281 ip-10-0-130-0 kubenswrapper[5998]: E1030 01:59:55.218243 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 01:59:55.941648 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:55.941617 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:56.541487 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:56.541448 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="925ff5f4-100e-46a4-a906-3a9ecef96765" path="/var/lib/kubelet/pods/925ff5f4-100e-46a4-a906-3a9ecef96765/volumes" Oct 30 01:59:57.751186 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:57.751150 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:59.252264 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:59.252230 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:59.857720 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:59.857677 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 01:59:59.978361 ip-10-0-130-0 kubenswrapper[5998]: I1030 01:59:59.978324 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:00.167150 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.167084 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65"] Oct 30 02:00:00.167764 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.167748 5998 cpu_manager.go:414] "containerMap: removing container" podUID="925ff5f4-100e-46a4-a906-3a9ecef96765" containerName="istio-proxy" Oct 30 02:00:00.167815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.167766 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="925ff5f4-100e-46a4-a906-3a9ecef96765" containerName="istio-proxy" Oct 30 02:00:00.167923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.167896 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="925ff5f4-100e-46a4-a906-3a9ecef96765" containerName="istio-proxy" Oct 30 02:00:00.178229 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.178186 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w"] Oct 30 02:00:00.182215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.182183 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" Oct 30 02:00:00.187210 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.187186 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g"] Oct 30 02:00:00.187454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.187282 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" Oct 30 02:00:00.192282 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.191961 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l"] Oct 30 02:00:00.192282 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.192178 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" Oct 30 02:00:00.198110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.198069 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl"] Oct 30 02:00:00.199872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.199846 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" Oct 30 02:00:00.207429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.207401 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:00:00.208374 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.208351 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:00:00.283432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.283383 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:00:00.287360 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.287327 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af80b61b-f6eb-4e60-abbd-83495acec7d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w\" (UID: \"af80b61b-f6eb-4e60-abbd-83495acec7d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" Oct 30 02:00:00.287522 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.287395 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48178a43-c541-48df-a3cd-5712a41998e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65\" (UID: \"48178a43-c541-48df-a3cd-5712a41998e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" Oct 30 02:00:00.287522 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.287452 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1008ca2d-680d-496b-acec-7ac087a0d48c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g\" (UID: \"1008ca2d-680d-496b-acec-7ac087a0d48c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" Oct 30 02:00:00.287642 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.287529 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghhb8\" (UniqueName: \"kubernetes.io/projected/ee325398-ca84-48b8-90b1-105e16fbebd4-kube-api-access-ghhb8\") pod \"maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l\" (UID: \"ee325398-ca84-48b8-90b1-105e16fbebd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" Oct 30 02:00:00.287642 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.287582 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv9w6\" (UniqueName: \"kubernetes.io/projected/1008ca2d-680d-496b-acec-7ac087a0d48c-kube-api-access-gv9w6\") pod \"maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g\" (UID: \"1008ca2d-680d-496b-acec-7ac087a0d48c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" Oct 30 02:00:00.287642 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.287629 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee325398-ca84-48b8-90b1-105e16fbebd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l\" (UID: \"ee325398-ca84-48b8-90b1-105e16fbebd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" Oct 30 02:00:00.288171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.287761 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwzzv\" (UniqueName: \"kubernetes.io/projected/48178a43-c541-48df-a3cd-5712a41998e9-kube-api-access-pwzzv\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65\" (UID: \"48178a43-c541-48df-a3cd-5712a41998e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" Oct 30 02:00:00.288171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.287879 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k6ms\" (UniqueName: \"kubernetes.io/projected/af80b61b-f6eb-4e60-abbd-83495acec7d9-kube-api-access-2k6ms\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w\" (UID: \"af80b61b-f6eb-4e60-abbd-83495acec7d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" Oct 30 02:00:00.296072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.296047 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65"] Oct 30 02:00:00.296383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.296369 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g"] Oct 30 02:00:00.296507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.296499 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l"] Oct 30 02:00:00.296598 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.296591 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5"] Oct 30 02:00:00.297248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.296325 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" Oct 30 02:00:00.309269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.307318 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5"] Oct 30 02:00:00.320173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.320081 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" Oct 30 02:00:00.357677 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.357629 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" Oct 30 02:00:00.364111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.363636 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x"] Oct 30 02:00:00.385863 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.385834 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5"] Oct 30 02:00:00.386185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.386166 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" Oct 30 02:00:00.389737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.389703 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv9w6\" (UniqueName: \"kubernetes.io/projected/1008ca2d-680d-496b-acec-7ac087a0d48c-kube-api-access-gv9w6\") pod \"maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g\" (UID: \"1008ca2d-680d-496b-acec-7ac087a0d48c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" Oct 30 02:00:00.389972 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.389931 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee325398-ca84-48b8-90b1-105e16fbebd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l\" (UID: \"ee325398-ca84-48b8-90b1-105e16fbebd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" Oct 30 02:00:00.390137 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.390123 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb8tp\" (UniqueName: \"kubernetes.io/projected/ffb61d79-7ce6-46d2-9f0d-712fb8a446a1-kube-api-access-tb8tp\") pod \"maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5\" (UID: \"ffb61d79-7ce6-46d2-9f0d-712fb8a446a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" Oct 30 02:00:00.390303 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.390277 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bq54\" (UniqueName: \"kubernetes.io/projected/847d96ae-df65-41f4-8663-53fdbd7a4b97-kube-api-access-4bq54\") pod \"maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5\" (UID: \"847d96ae-df65-41f4-8663-53fdbd7a4b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" Oct 30 02:00:00.390509 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.390492 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwzzv\" (UniqueName: \"kubernetes.io/projected/48178a43-c541-48df-a3cd-5712a41998e9-kube-api-access-pwzzv\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65\" (UID: \"48178a43-c541-48df-a3cd-5712a41998e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" Oct 30 02:00:00.390727 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.390710 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6132fae-af61-4920-b993-8eb4760609ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888760-rxmvl\" (UID: \"a6132fae-af61-4920-b993-8eb4760609ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" Oct 30 02:00:00.390869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.390855 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k6ms\" (UniqueName: \"kubernetes.io/projected/af80b61b-f6eb-4e60-abbd-83495acec7d9-kube-api-access-2k6ms\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w\" (UID: \"af80b61b-f6eb-4e60-abbd-83495acec7d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" Oct 30 02:00:00.390994 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.390980 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb61d79-7ce6-46d2-9f0d-712fb8a446a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5\" (UID: \"ffb61d79-7ce6-46d2-9f0d-712fb8a446a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" Oct 30 02:00:00.391151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.391136 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af80b61b-f6eb-4e60-abbd-83495acec7d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w\" (UID: \"af80b61b-f6eb-4e60-abbd-83495acec7d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" Oct 30 02:00:00.392766 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.392729 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4sjq\" (UniqueName: \"kubernetes.io/projected/a6132fae-af61-4920-b993-8eb4760609ff-kube-api-access-w4sjq\") pod \"maas-api-key-cleanup-29888760-rxmvl\" (UID: \"a6132fae-af61-4920-b993-8eb4760609ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" Oct 30 02:00:00.396737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.395472 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48178a43-c541-48df-a3cd-5712a41998e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65\" (UID: \"48178a43-c541-48df-a3cd-5712a41998e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" Oct 30 02:00:00.396737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.395546 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1008ca2d-680d-496b-acec-7ac087a0d48c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g\" (UID: \"1008ca2d-680d-496b-acec-7ac087a0d48c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" Oct 30 02:00:00.396737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.395618 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/847d96ae-df65-41f4-8663-53fdbd7a4b97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5\" (UID: \"847d96ae-df65-41f4-8663-53fdbd7a4b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" Oct 30 02:00:00.396737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.395655 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghhb8\" (UniqueName: \"kubernetes.io/projected/ee325398-ca84-48b8-90b1-105e16fbebd4-kube-api-access-ghhb8\") pod \"maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l\" (UID: \"ee325398-ca84-48b8-90b1-105e16fbebd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" Oct 30 02:00:00.396737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.395216 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af80b61b-f6eb-4e60-abbd-83495acec7d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w\" (UID: \"af80b61b-f6eb-4e60-abbd-83495acec7d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" Oct 30 02:00:00.396737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.395941 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee325398-ca84-48b8-90b1-105e16fbebd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l\" (UID: \"ee325398-ca84-48b8-90b1-105e16fbebd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" Oct 30 02:00:00.396737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.396591 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48178a43-c541-48df-a3cd-5712a41998e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65\" (UID: \"48178a43-c541-48df-a3cd-5712a41998e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" Oct 30 02:00:00.396737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.396626 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1008ca2d-680d-496b-acec-7ac087a0d48c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g\" (UID: \"1008ca2d-680d-496b-acec-7ac087a0d48c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" Oct 30 02:00:00.431344 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.418178 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w"] Oct 30 02:00:00.431344 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.423035 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k6ms\" (UniqueName: \"kubernetes.io/projected/af80b61b-f6eb-4e60-abbd-83495acec7d9-kube-api-access-2k6ms\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w\" (UID: \"af80b61b-f6eb-4e60-abbd-83495acec7d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" Oct 30 02:00:00.431344 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.423511 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwzzv\" (UniqueName: \"kubernetes.io/projected/48178a43-c541-48df-a3cd-5712a41998e9-kube-api-access-pwzzv\") pod \"maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65\" (UID: \"48178a43-c541-48df-a3cd-5712a41998e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" Oct 30 02:00:00.431344 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.426441 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv9w6\" (UniqueName: \"kubernetes.io/projected/1008ca2d-680d-496b-acec-7ac087a0d48c-kube-api-access-gv9w6\") pod \"maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g\" (UID: \"1008ca2d-680d-496b-acec-7ac087a0d48c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" Oct 30 02:00:00.431344 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.429651 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghhb8\" (UniqueName: \"kubernetes.io/projected/ee325398-ca84-48b8-90b1-105e16fbebd4-kube-api-access-ghhb8\") pod \"maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l\" (UID: \"ee325398-ca84-48b8-90b1-105e16fbebd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" Oct 30 02:00:00.468829 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.468483 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:00.469256 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.469179 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl"] Oct 30 02:00:00.479011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.478875 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5"] Oct 30 02:00:00.494355 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.494286 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x"] Oct 30 02:00:00.498079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.498043 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/847d96ae-df65-41f4-8663-53fdbd7a4b97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5\" (UID: \"847d96ae-df65-41f4-8663-53fdbd7a4b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" Oct 30 02:00:00.498244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.498137 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7w87\" (UniqueName: \"kubernetes.io/projected/1c0254eb-8332-4172-b1ba-a8235aa882a8-kube-api-access-h7w87\") pod \"maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x\" (UID: \"1c0254eb-8332-4172-b1ba-a8235aa882a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" Oct 30 02:00:00.498312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.498252 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb8tp\" (UniqueName: \"kubernetes.io/projected/ffb61d79-7ce6-46d2-9f0d-712fb8a446a1-kube-api-access-tb8tp\") pod \"maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5\" (UID: \"ffb61d79-7ce6-46d2-9f0d-712fb8a446a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" Oct 30 02:00:00.498377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.498309 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bq54\" (UniqueName: \"kubernetes.io/projected/847d96ae-df65-41f4-8663-53fdbd7a4b97-kube-api-access-4bq54\") pod \"maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5\" (UID: \"847d96ae-df65-41f4-8663-53fdbd7a4b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" Oct 30 02:00:00.498563 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.498401 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c0254eb-8332-4172-b1ba-a8235aa882a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x\" (UID: \"1c0254eb-8332-4172-b1ba-a8235aa882a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" Oct 30 02:00:00.498563 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.498514 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6132fae-af61-4920-b993-8eb4760609ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888760-rxmvl\" (UID: \"a6132fae-af61-4920-b993-8eb4760609ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" Oct 30 02:00:00.498673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.498583 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb61d79-7ce6-46d2-9f0d-712fb8a446a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5\" (UID: \"ffb61d79-7ce6-46d2-9f0d-712fb8a446a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" Oct 30 02:00:00.498673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.498654 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4sjq\" (UniqueName: \"kubernetes.io/projected/a6132fae-af61-4920-b993-8eb4760609ff-kube-api-access-w4sjq\") pod \"maas-api-key-cleanup-29888760-rxmvl\" (UID: \"a6132fae-af61-4920-b993-8eb4760609ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" Oct 30 02:00:00.504134 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.501467 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb61d79-7ce6-46d2-9f0d-712fb8a446a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5\" (UID: \"ffb61d79-7ce6-46d2-9f0d-712fb8a446a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" Oct 30 02:00:00.516443 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.512693 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6132fae-af61-4920-b993-8eb4760609ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888760-rxmvl\" (UID: \"a6132fae-af61-4920-b993-8eb4760609ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" Oct 30 02:00:00.537289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.537229 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/847d96ae-df65-41f4-8663-53fdbd7a4b97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5\" (UID: \"847d96ae-df65-41f4-8663-53fdbd7a4b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" Oct 30 02:00:00.558939 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.556768 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb8tp\" (UniqueName: \"kubernetes.io/projected/ffb61d79-7ce6-46d2-9f0d-712fb8a446a1-kube-api-access-tb8tp\") pod \"maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5\" (UID: \"ffb61d79-7ce6-46d2-9f0d-712fb8a446a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" Oct 30 02:00:00.558939 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.557324 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bq54\" (UniqueName: \"kubernetes.io/projected/847d96ae-df65-41f4-8663-53fdbd7a4b97-kube-api-access-4bq54\") pod \"maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5\" (UID: \"847d96ae-df65-41f4-8663-53fdbd7a4b97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" Oct 30 02:00:00.560982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.560916 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4sjq\" (UniqueName: \"kubernetes.io/projected/a6132fae-af61-4920-b993-8eb4760609ff-kube-api-access-w4sjq\") pod \"maas-api-key-cleanup-29888760-rxmvl\" (UID: \"a6132fae-af61-4920-b993-8eb4760609ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" Oct 30 02:00:00.585143 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.584525 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" Oct 30 02:00:00.607166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.594283 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:00.607166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.605530 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7w87\" (UniqueName: \"kubernetes.io/projected/1c0254eb-8332-4172-b1ba-a8235aa882a8-kube-api-access-h7w87\") pod \"maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x\" (UID: \"1c0254eb-8332-4172-b1ba-a8235aa882a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" Oct 30 02:00:00.637977 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.637935 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c0254eb-8332-4172-b1ba-a8235aa882a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x\" (UID: \"1c0254eb-8332-4172-b1ba-a8235aa882a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" Oct 30 02:00:00.649652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.644634 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" Oct 30 02:00:00.718304 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.718264 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" Oct 30 02:00:00.726383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.726339 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" Oct 30 02:00:00.759526 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.759489 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c0254eb-8332-4172-b1ba-a8235aa882a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x\" (UID: \"1c0254eb-8332-4172-b1ba-a8235aa882a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" Oct 30 02:00:00.784765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.768459 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" Oct 30 02:00:00.784765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.782417 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" Oct 30 02:00:00.805154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.801343 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" Oct 30 02:00:00.933719 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:00.933448 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7w87\" (UniqueName: \"kubernetes.io/projected/1c0254eb-8332-4172-b1ba-a8235aa882a8-kube-api-access-h7w87\") pod \"maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x\" (UID: \"1c0254eb-8332-4172-b1ba-a8235aa882a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" Oct 30 02:00:01.214166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:01.188038 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" Oct 30 02:00:01.550306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:01.550258 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w"] Oct 30 02:00:01.560699 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:00:01.560655 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf80b61b_f6eb_4e60_abbd_83495acec7d9.slice/crio-948025b5276e69d76050364ef3b0d815ba0a7e218310b68fda5d877541fdce8f WatchSource:0}: Error finding container 948025b5276e69d76050364ef3b0d815ba0a7e218310b68fda5d877541fdce8f: Status 404 returned error can't find the container with id 948025b5276e69d76050364ef3b0d815ba0a7e218310b68fda5d877541fdce8f Oct 30 02:00:01.751133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:01.750575 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:01.961915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:01.961800 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l"] Oct 30 02:00:01.968524 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:01.968443 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5"] Oct 30 02:00:01.993123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:01.987162 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g"] Oct 30 02:00:01.993422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:01.993370 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65"] Oct 30 02:00:02.070039 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:02.069930 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" event={"ID":"847d96ae-df65-41f4-8663-53fdbd7a4b97","Type":"ContainerStarted","Data":"819f6c425c7bc20989f5c535068fa26e9d7392fd7ccd7bd55d9c729f1b1349fd"} Oct 30 02:00:02.074061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:02.073786 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" event={"ID":"ee325398-ca84-48b8-90b1-105e16fbebd4","Type":"ContainerStarted","Data":"533f53c6039cff8c194fdec79253267cee8d6a883f9a4c3d26a9249c8e8df816"} Oct 30 02:00:02.077692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:02.077647 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" event={"ID":"af80b61b-f6eb-4e60-abbd-83495acec7d9","Type":"ContainerStarted","Data":"948025b5276e69d76050364ef3b0d815ba0a7e218310b68fda5d877541fdce8f"} Oct 30 02:00:02.160137 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:02.158263 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5"] Oct 30 02:00:02.280804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:02.280555 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x"] Oct 30 02:00:02.288294 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:02.285410 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl"] Oct 30 02:00:02.288294 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:00:02.287566 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0254eb_8332_4172_b1ba_a8235aa882a8.slice/crio-cb0b9422003fe76ebadf783f00128fd8fc8ceff72abaddd2af9e1da734f1c2f1 WatchSource:0}: Error finding container cb0b9422003fe76ebadf783f00128fd8fc8ceff72abaddd2af9e1da734f1c2f1: Status 404 returned error can't find the container with id cb0b9422003fe76ebadf783f00128fd8fc8ceff72abaddd2af9e1da734f1c2f1 Oct 30 02:00:02.946521 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:02.945165 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:03.274123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.262154 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:03.296123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.286575 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" event={"ID":"48178a43-c541-48df-a3cd-5712a41998e9","Type":"ContainerStarted","Data":"f5ad71fbb8d4ff94e7c356d7b92967056c393a27368f8fd825797ac59f2ad8b1"} Oct 30 02:00:03.296123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.286644 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" event={"ID":"48178a43-c541-48df-a3cd-5712a41998e9","Type":"ContainerStarted","Data":"efb73f1a7bd597e8a59b3cd7123d880807268f69d420a8c33554da23572da778"} Oct 30 02:00:03.296123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.291954 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" event={"ID":"847d96ae-df65-41f4-8663-53fdbd7a4b97","Type":"ContainerStarted","Data":"a5591cde90ba19adeafda50464f242edb93ce76ac46730d9ff2fb9952a28af17"} Oct 30 02:00:03.300324 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.299874 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" event={"ID":"1c0254eb-8332-4172-b1ba-a8235aa882a8","Type":"ContainerStarted","Data":"195601dd0fafdfd2066d92c71b74772f676c204e7dc00812f76498d886e0ca76"} Oct 30 02:00:03.306275 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.306222 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" event={"ID":"1c0254eb-8332-4172-b1ba-a8235aa882a8","Type":"ContainerStarted","Data":"cb0b9422003fe76ebadf783f00128fd8fc8ceff72abaddd2af9e1da734f1c2f1"} Oct 30 02:00:03.411111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.407724 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" event={"ID":"ee325398-ca84-48b8-90b1-105e16fbebd4","Type":"ContainerStarted","Data":"85327c16ffe7fe0a0c51835ba64ad09e4be48a726216ec1929847fab7ab1392d"} Oct 30 02:00:03.443123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.437013 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" podStartSLOduration=3.436991848 podStartE2EDuration="3.436991848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:03.4022699 +0000 UTC m=+1832.981060982" watchObservedRunningTime="2026-10-30 02:00:03.436991848 +0000 UTC m=+1833.015782932" Oct 30 02:00:03.467200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.462246 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" event={"ID":"ffb61d79-7ce6-46d2-9f0d-712fb8a446a1","Type":"ContainerStarted","Data":"279111b9020a0e72bd04926dde27a5b0988e7f6936aa9672348a0b1f10f098d5"} Oct 30 02:00:03.467200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.462313 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" event={"ID":"ffb61d79-7ce6-46d2-9f0d-712fb8a446a1","Type":"ContainerStarted","Data":"44be394eeea54e18bceb4c15fb69cc08e34a6895d384b4128081e51c9801c248"} Oct 30 02:00:03.651634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.650329 5998 generic.go:356] "Generic (PLEG): container finished" podID="af80b61b-f6eb-4e60-abbd-83495acec7d9" containerID="0012d97864b0c211b04b7c389e5cbef86b551b5236b7eb7c3da93e96ee6022de" exitCode=60 Oct 30 02:00:03.651634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.650469 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" event={"ID":"af80b61b-f6eb-4e60-abbd-83495acec7d9","Type":"ContainerDied","Data":"0012d97864b0c211b04b7c389e5cbef86b551b5236b7eb7c3da93e96ee6022de"} Oct 30 02:00:03.651634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.651000 5998 scope.go:122] "RemoveContainer" containerID="0012d97864b0c211b04b7c389e5cbef86b551b5236b7eb7c3da93e96ee6022de" Oct 30 02:00:03.675963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.663079 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" event={"ID":"1008ca2d-680d-496b-acec-7ac087a0d48c","Type":"ContainerStarted","Data":"6852fd6f7eb7d19371f6863c4c04818249507a9fc8c6efcff826a42fea46e69a"} Oct 30 02:00:03.675963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.663163 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" event={"ID":"1008ca2d-680d-496b-acec-7ac087a0d48c","Type":"ContainerStarted","Data":"1d36c2e99fe49b9347c255c7742bee10506c7fb5ae95e1603f9ede3a3acae39c"} Oct 30 02:00:03.681829 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.680976 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" podStartSLOduration=3.68095557 podStartE2EDuration="3.68095557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:03.459529317 +0000 UTC m=+1833.038320402" watchObservedRunningTime="2026-10-30 02:00:03.68095557 +0000 UTC m=+1833.259746653" Oct 30 02:00:03.685178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.683702 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" podStartSLOduration=3.683682562 podStartE2EDuration="3.683682562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:03.649356052 +0000 UTC m=+1833.228147134" watchObservedRunningTime="2026-10-30 02:00:03.683682562 +0000 UTC m=+1833.262473644" Oct 30 02:00:03.686373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.686013 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" event={"ID":"a6132fae-af61-4920-b993-8eb4760609ff","Type":"ContainerStarted","Data":"25ba1af636b6acefd25cb948d125cab8f0fd2f45584b27c0b7fe101c583ca59c"} Oct 30 02:00:03.686373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.686073 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" event={"ID":"a6132fae-af61-4920-b993-8eb4760609ff","Type":"ContainerStarted","Data":"90138c9c56ed876807b21d2e6b1ccb1e92565c788a3d991d614eed5bb289fede"} Oct 30 02:00:03.695367 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.692906 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" podStartSLOduration=3.692887991 podStartE2EDuration="3.692887991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:03.688229557 +0000 UTC m=+1833.267020640" watchObservedRunningTime="2026-10-30 02:00:03.692887991 +0000 UTC m=+1833.271679073" Oct 30 02:00:03.746612 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.746520 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" podStartSLOduration=3.746496055 podStartE2EDuration="3.746496055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:03.741837764 +0000 UTC m=+1833.320628878" watchObservedRunningTime="2026-10-30 02:00:03.746496055 +0000 UTC m=+1833.325287137" Oct 30 02:00:03.783177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.783060 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" podStartSLOduration=3.783041693 podStartE2EDuration="3.783041693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:03.780791959 +0000 UTC m=+1833.359583051" watchObservedRunningTime="2026-10-30 02:00:03.783041693 +0000 UTC m=+1833.361832774" Oct 30 02:00:03.820565 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:03.820515 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" podStartSLOduration=3.820498556 podStartE2EDuration="3.820498556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:03.819448407 +0000 UTC m=+1833.398239489" watchObservedRunningTime="2026-10-30 02:00:03.820498556 +0000 UTC m=+1833.399289636" Oct 30 02:00:04.122810 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.122704 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:04.695882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.695843 5998 generic.go:356] "Generic (PLEG): container finished" podID="a6132fae-af61-4920-b993-8eb4760609ff" containerID="25ba1af636b6acefd25cb948d125cab8f0fd2f45584b27c0b7fe101c583ca59c" exitCode=60 Oct 30 02:00:04.696069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.695927 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" event={"ID":"a6132fae-af61-4920-b993-8eb4760609ff","Type":"ContainerDied","Data":"25ba1af636b6acefd25cb948d125cab8f0fd2f45584b27c0b7fe101c583ca59c"} Oct 30 02:00:04.696307 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.696290 5998 scope.go:122] "RemoveContainer" containerID="25ba1af636b6acefd25cb948d125cab8f0fd2f45584b27c0b7fe101c583ca59c" Oct 30 02:00:04.697815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.697790 5998 generic.go:356] "Generic (PLEG): container finished" podID="48178a43-c541-48df-a3cd-5712a41998e9" containerID="f5ad71fbb8d4ff94e7c356d7b92967056c393a27368f8fd825797ac59f2ad8b1" exitCode=60 Oct 30 02:00:04.697898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.697868 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" event={"ID":"48178a43-c541-48df-a3cd-5712a41998e9","Type":"ContainerDied","Data":"f5ad71fbb8d4ff94e7c356d7b92967056c393a27368f8fd825797ac59f2ad8b1"} Oct 30 02:00:04.698138 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.698085 5998 scope.go:122] "RemoveContainer" containerID="f5ad71fbb8d4ff94e7c356d7b92967056c393a27368f8fd825797ac59f2ad8b1" Oct 30 02:00:04.699857 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.699830 5998 generic.go:356] "Generic (PLEG): container finished" podID="847d96ae-df65-41f4-8663-53fdbd7a4b97" containerID="a5591cde90ba19adeafda50464f242edb93ce76ac46730d9ff2fb9952a28af17" exitCode=60 Oct 30 02:00:04.699934 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.699860 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" event={"ID":"847d96ae-df65-41f4-8663-53fdbd7a4b97","Type":"ContainerDied","Data":"a5591cde90ba19adeafda50464f242edb93ce76ac46730d9ff2fb9952a28af17"} Oct 30 02:00:04.700205 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.700187 5998 scope.go:122] "RemoveContainer" containerID="a5591cde90ba19adeafda50464f242edb93ce76ac46730d9ff2fb9952a28af17" Oct 30 02:00:04.708608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.707901 5998 generic.go:356] "Generic (PLEG): container finished" podID="1c0254eb-8332-4172-b1ba-a8235aa882a8" containerID="195601dd0fafdfd2066d92c71b74772f676c204e7dc00812f76498d886e0ca76" exitCode=60 Oct 30 02:00:04.708608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.708114 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" event={"ID":"1c0254eb-8332-4172-b1ba-a8235aa882a8","Type":"ContainerDied","Data":"195601dd0fafdfd2066d92c71b74772f676c204e7dc00812f76498d886e0ca76"} Oct 30 02:00:04.708608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.708365 5998 scope.go:122] "RemoveContainer" containerID="195601dd0fafdfd2066d92c71b74772f676c204e7dc00812f76498d886e0ca76" Oct 30 02:00:04.711971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.711946 5998 generic.go:356] "Generic (PLEG): container finished" podID="ee325398-ca84-48b8-90b1-105e16fbebd4" containerID="85327c16ffe7fe0a0c51835ba64ad09e4be48a726216ec1929847fab7ab1392d" exitCode=60 Oct 30 02:00:04.712185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.712019 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" event={"ID":"ee325398-ca84-48b8-90b1-105e16fbebd4","Type":"ContainerDied","Data":"85327c16ffe7fe0a0c51835ba64ad09e4be48a726216ec1929847fab7ab1392d"} Oct 30 02:00:04.713238 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.713220 5998 scope.go:122] "RemoveContainer" containerID="85327c16ffe7fe0a0c51835ba64ad09e4be48a726216ec1929847fab7ab1392d" Oct 30 02:00:04.718970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.718943 5998 generic.go:356] "Generic (PLEG): container finished" podID="ffb61d79-7ce6-46d2-9f0d-712fb8a446a1" containerID="279111b9020a0e72bd04926dde27a5b0988e7f6936aa9672348a0b1f10f098d5" exitCode=60 Oct 30 02:00:04.719084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.719039 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" event={"ID":"ffb61d79-7ce6-46d2-9f0d-712fb8a446a1","Type":"ContainerDied","Data":"279111b9020a0e72bd04926dde27a5b0988e7f6936aa9672348a0b1f10f098d5"} Oct 30 02:00:04.719756 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.719733 5998 scope.go:122] "RemoveContainer" containerID="279111b9020a0e72bd04926dde27a5b0988e7f6936aa9672348a0b1f10f098d5" Oct 30 02:00:04.730836 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.730685 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" event={"ID":"af80b61b-f6eb-4e60-abbd-83495acec7d9","Type":"ContainerStarted","Data":"3d4281d8cc242d23bd8f9bb0a088fdeab41a8cd5e4e4ccd6b62c9e433c6ab607"} Oct 30 02:00:04.733542 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.733514 5998 generic.go:356] "Generic (PLEG): container finished" podID="1008ca2d-680d-496b-acec-7ac087a0d48c" containerID="6852fd6f7eb7d19371f6863c4c04818249507a9fc8c6efcff826a42fea46e69a" exitCode=60 Oct 30 02:00:04.733656 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.733562 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" event={"ID":"1008ca2d-680d-496b-acec-7ac087a0d48c","Type":"ContainerDied","Data":"6852fd6f7eb7d19371f6863c4c04818249507a9fc8c6efcff826a42fea46e69a"} Oct 30 02:00:04.733876 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.733852 5998 scope.go:122] "RemoveContainer" containerID="6852fd6f7eb7d19371f6863c4c04818249507a9fc8c6efcff826a42fea46e69a" Oct 30 02:00:04.993340 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:04.993227 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" podStartSLOduration=4.993206367 podStartE2EDuration="4.993206367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:04.992889074 +0000 UTC m=+1834.571680166" watchObservedRunningTime="2026-10-30 02:00:04.993206367 +0000 UTC m=+1834.571997447" Oct 30 02:00:05.749922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:05.749884 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" event={"ID":"847d96ae-df65-41f4-8663-53fdbd7a4b97","Type":"ContainerStarted","Data":"ee205f95dc53de6f75096ef8d24135db5a5dc1db844a539ea49d55525f56d5f5"} Oct 30 02:00:05.752623 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:05.752588 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" event={"ID":"1c0254eb-8332-4172-b1ba-a8235aa882a8","Type":"ContainerStarted","Data":"2c1129429b2941151f228cb46424240693839f2c1d8f52e0a86e908b4fcb3b32"} Oct 30 02:00:05.755649 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:05.755577 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" event={"ID":"ee325398-ca84-48b8-90b1-105e16fbebd4","Type":"ContainerStarted","Data":"a9cd4e29bb89a8d810cecaf945899ee6a46f8d49d8b18bd9b8ce91d4b95f761f"} Oct 30 02:00:05.758239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:05.758205 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" event={"ID":"ffb61d79-7ce6-46d2-9f0d-712fb8a446a1","Type":"ContainerStarted","Data":"20d86d9ae43c284737d85da28db97db489911d269cca09f9ac1ddd8e652adc06"} Oct 30 02:00:05.760724 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:05.760698 5998 generic.go:356] "Generic (PLEG): container finished" podID="af80b61b-f6eb-4e60-abbd-83495acec7d9" containerID="3d4281d8cc242d23bd8f9bb0a088fdeab41a8cd5e4e4ccd6b62c9e433c6ab607" exitCode=60 Oct 30 02:00:05.760864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:05.760780 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" event={"ID":"af80b61b-f6eb-4e60-abbd-83495acec7d9","Type":"ContainerDied","Data":"3d4281d8cc242d23bd8f9bb0a088fdeab41a8cd5e4e4ccd6b62c9e433c6ab607"} Oct 30 02:00:05.760864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:05.760838 5998 scope.go:122] "RemoveContainer" containerID="0012d97864b0c211b04b7c389e5cbef86b551b5236b7eb7c3da93e96ee6022de" Oct 30 02:00:05.761126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:05.761065 5998 scope.go:122] "RemoveContainer" containerID="3d4281d8cc242d23bd8f9bb0a088fdeab41a8cd5e4e4ccd6b62c9e433c6ab607" Oct 30 02:00:05.761524 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:05.761497 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w_odh-ai-gateway-infra(af80b61b-f6eb-4e60-abbd-83495acec7d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" podUID="af80b61b-f6eb-4e60-abbd-83495acec7d9" Oct 30 02:00:05.763431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:05.763405 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" event={"ID":"1008ca2d-680d-496b-acec-7ac087a0d48c","Type":"ContainerStarted","Data":"50732e69cb41ce09869a4770965b0f4dbedc30e5f902ef27ec801cbd7297c95c"} Oct 30 02:00:05.765613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:05.765591 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" event={"ID":"a6132fae-af61-4920-b993-8eb4760609ff","Type":"ContainerStarted","Data":"7ebb63bae98970cc0bdba9b4f35e306a36cfc7e758f0fd067ae46aa8f7c77e1e"} Oct 30 02:00:05.768000 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:05.767978 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" event={"ID":"48178a43-c541-48df-a3cd-5712a41998e9","Type":"ContainerStarted","Data":"b2719d68be32fd7c8d9d94db29884973a015b1d9048ed0ad0d8788697a55a9ed"} Oct 30 02:00:05.786967 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:05.786934 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:05.930144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:05.930051 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:06.779581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.779528 5998 generic.go:356] "Generic (PLEG): container finished" podID="847d96ae-df65-41f4-8663-53fdbd7a4b97" containerID="ee205f95dc53de6f75096ef8d24135db5a5dc1db844a539ea49d55525f56d5f5" exitCode=60 Oct 30 02:00:06.780014 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.779606 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" event={"ID":"847d96ae-df65-41f4-8663-53fdbd7a4b97","Type":"ContainerDied","Data":"ee205f95dc53de6f75096ef8d24135db5a5dc1db844a539ea49d55525f56d5f5"} Oct 30 02:00:06.780014 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.779680 5998 scope.go:122] "RemoveContainer" containerID="a5591cde90ba19adeafda50464f242edb93ce76ac46730d9ff2fb9952a28af17" Oct 30 02:00:06.780014 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.779956 5998 scope.go:122] "RemoveContainer" containerID="ee205f95dc53de6f75096ef8d24135db5a5dc1db844a539ea49d55525f56d5f5" Oct 30 02:00:06.780422 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:06.780388 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5_odh-ai-gateway-infra(847d96ae-df65-41f4-8663-53fdbd7a4b97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" podUID="847d96ae-df65-41f4-8663-53fdbd7a4b97" Oct 30 02:00:06.782783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.782757 5998 generic.go:356] "Generic (PLEG): container finished" podID="1c0254eb-8332-4172-b1ba-a8235aa882a8" containerID="2c1129429b2941151f228cb46424240693839f2c1d8f52e0a86e908b4fcb3b32" exitCode=60 Oct 30 02:00:06.782883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.782841 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" event={"ID":"1c0254eb-8332-4172-b1ba-a8235aa882a8","Type":"ContainerDied","Data":"2c1129429b2941151f228cb46424240693839f2c1d8f52e0a86e908b4fcb3b32"} Oct 30 02:00:06.783174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.783155 5998 scope.go:122] "RemoveContainer" containerID="2c1129429b2941151f228cb46424240693839f2c1d8f52e0a86e908b4fcb3b32" Oct 30 02:00:06.783548 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:06.783475 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x_odh-ai-gateway-infra(1c0254eb-8332-4172-b1ba-a8235aa882a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" podUID="1c0254eb-8332-4172-b1ba-a8235aa882a8" Oct 30 02:00:06.785360 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.785331 5998 generic.go:356] "Generic (PLEG): container finished" podID="ee325398-ca84-48b8-90b1-105e16fbebd4" containerID="a9cd4e29bb89a8d810cecaf945899ee6a46f8d49d8b18bd9b8ce91d4b95f761f" exitCode=60 Oct 30 02:00:06.785471 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.785359 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" event={"ID":"ee325398-ca84-48b8-90b1-105e16fbebd4","Type":"ContainerDied","Data":"a9cd4e29bb89a8d810cecaf945899ee6a46f8d49d8b18bd9b8ce91d4b95f761f"} Oct 30 02:00:06.785802 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.785781 5998 scope.go:122] "RemoveContainer" containerID="a9cd4e29bb89a8d810cecaf945899ee6a46f8d49d8b18bd9b8ce91d4b95f761f" Oct 30 02:00:06.786184 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:06.786156 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l_odh-ai-gateway-infra(ee325398-ca84-48b8-90b1-105e16fbebd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" podUID="ee325398-ca84-48b8-90b1-105e16fbebd4" Oct 30 02:00:06.787502 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.787477 5998 generic.go:356] "Generic (PLEG): container finished" podID="ffb61d79-7ce6-46d2-9f0d-712fb8a446a1" containerID="20d86d9ae43c284737d85da28db97db489911d269cca09f9ac1ddd8e652adc06" exitCode=60 Oct 30 02:00:06.787600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.787566 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" event={"ID":"ffb61d79-7ce6-46d2-9f0d-712fb8a446a1","Type":"ContainerDied","Data":"20d86d9ae43c284737d85da28db97db489911d269cca09f9ac1ddd8e652adc06"} Oct 30 02:00:06.787814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.787797 5998 scope.go:122] "RemoveContainer" containerID="20d86d9ae43c284737d85da28db97db489911d269cca09f9ac1ddd8e652adc06" Oct 30 02:00:06.788125 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:06.788064 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5_odh-ai-gateway-infra(ffb61d79-7ce6-46d2-9f0d-712fb8a446a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" podUID="ffb61d79-7ce6-46d2-9f0d-712fb8a446a1" Oct 30 02:00:06.789753 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.789735 5998 scope.go:122] "RemoveContainer" containerID="3d4281d8cc242d23bd8f9bb0a088fdeab41a8cd5e4e4ccd6b62c9e433c6ab607" Oct 30 02:00:06.790029 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:06.790007 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w_odh-ai-gateway-infra(af80b61b-f6eb-4e60-abbd-83495acec7d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" podUID="af80b61b-f6eb-4e60-abbd-83495acec7d9" Oct 30 02:00:06.791618 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.791594 5998 generic.go:356] "Generic (PLEG): container finished" podID="1008ca2d-680d-496b-acec-7ac087a0d48c" containerID="50732e69cb41ce09869a4770965b0f4dbedc30e5f902ef27ec801cbd7297c95c" exitCode=60 Oct 30 02:00:06.791714 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.791684 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" event={"ID":"1008ca2d-680d-496b-acec-7ac087a0d48c","Type":"ContainerDied","Data":"50732e69cb41ce09869a4770965b0f4dbedc30e5f902ef27ec801cbd7297c95c"} Oct 30 02:00:06.791891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.791874 5998 scope.go:122] "RemoveContainer" containerID="50732e69cb41ce09869a4770965b0f4dbedc30e5f902ef27ec801cbd7297c95c" Oct 30 02:00:06.792161 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:06.792141 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g_odh-ai-gateway-infra(1008ca2d-680d-496b-acec-7ac087a0d48c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" podUID="1008ca2d-680d-496b-acec-7ac087a0d48c" Oct 30 02:00:06.793962 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.793936 5998 generic.go:356] "Generic (PLEG): container finished" podID="a6132fae-af61-4920-b993-8eb4760609ff" containerID="7ebb63bae98970cc0bdba9b4f35e306a36cfc7e758f0fd067ae46aa8f7c77e1e" exitCode=60 Oct 30 02:00:06.794050 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.794014 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" event={"ID":"a6132fae-af61-4920-b993-8eb4760609ff","Type":"ContainerDied","Data":"7ebb63bae98970cc0bdba9b4f35e306a36cfc7e758f0fd067ae46aa8f7c77e1e"} Oct 30 02:00:06.794409 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.794389 5998 scope.go:122] "RemoveContainer" containerID="7ebb63bae98970cc0bdba9b4f35e306a36cfc7e758f0fd067ae46aa8f7c77e1e" Oct 30 02:00:06.794704 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:06.794682 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888760-rxmvl_odh-ai-gateway-infra(a6132fae-af61-4920-b993-8eb4760609ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" podUID="a6132fae-af61-4920-b993-8eb4760609ff" Oct 30 02:00:06.795938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.795920 5998 scope.go:122] "RemoveContainer" containerID="195601dd0fafdfd2066d92c71b74772f676c204e7dc00812f76498d886e0ca76" Oct 30 02:00:06.796184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.796159 5998 generic.go:356] "Generic (PLEG): container finished" podID="48178a43-c541-48df-a3cd-5712a41998e9" containerID="b2719d68be32fd7c8d9d94db29884973a015b1d9048ed0ad0d8788697a55a9ed" exitCode=60 Oct 30 02:00:06.796273 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.796192 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" event={"ID":"48178a43-c541-48df-a3cd-5712a41998e9","Type":"ContainerDied","Data":"b2719d68be32fd7c8d9d94db29884973a015b1d9048ed0ad0d8788697a55a9ed"} Oct 30 02:00:06.796477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.796460 5998 scope.go:122] "RemoveContainer" containerID="b2719d68be32fd7c8d9d94db29884973a015b1d9048ed0ad0d8788697a55a9ed" Oct 30 02:00:06.796703 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:06.796682 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65_odh-ai-gateway-infra(48178a43-c541-48df-a3cd-5712a41998e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" podUID="48178a43-c541-48df-a3cd-5712a41998e9" Oct 30 02:00:06.810954 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.810927 5998 scope.go:122] "RemoveContainer" containerID="85327c16ffe7fe0a0c51835ba64ad09e4be48a726216ec1929847fab7ab1392d" Oct 30 02:00:06.827796 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.827765 5998 scope.go:122] "RemoveContainer" containerID="279111b9020a0e72bd04926dde27a5b0988e7f6936aa9672348a0b1f10f098d5" Oct 30 02:00:06.846974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.846952 5998 scope.go:122] "RemoveContainer" containerID="6852fd6f7eb7d19371f6863c4c04818249507a9fc8c6efcff826a42fea46e69a" Oct 30 02:00:06.867080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.866605 5998 scope.go:122] "RemoveContainer" containerID="25ba1af636b6acefd25cb948d125cab8f0fd2f45584b27c0b7fe101c583ca59c" Oct 30 02:00:06.887303 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.887272 5998 scope.go:122] "RemoveContainer" containerID="f5ad71fbb8d4ff94e7c356d7b92967056c393a27368f8fd825797ac59f2ad8b1" Oct 30 02:00:06.953640 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:06.953600 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:07.157055 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:07.156958 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:00:07.736916 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:07.736879 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:07.809061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:07.809019 5998 scope.go:122] "RemoveContainer" containerID="ee205f95dc53de6f75096ef8d24135db5a5dc1db844a539ea49d55525f56d5f5" Oct 30 02:00:07.809543 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:07.809446 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5_odh-ai-gateway-infra(847d96ae-df65-41f4-8663-53fdbd7a4b97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" podUID="847d96ae-df65-41f4-8663-53fdbd7a4b97" Oct 30 02:00:07.811064 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:07.811037 5998 scope.go:122] "RemoveContainer" containerID="2c1129429b2941151f228cb46424240693839f2c1d8f52e0a86e908b4fcb3b32" Oct 30 02:00:07.811499 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:07.811476 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x_odh-ai-gateway-infra(1c0254eb-8332-4172-b1ba-a8235aa882a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" podUID="1c0254eb-8332-4172-b1ba-a8235aa882a8" Oct 30 02:00:07.814583 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:07.814561 5998 scope.go:122] "RemoveContainer" containerID="a9cd4e29bb89a8d810cecaf945899ee6a46f8d49d8b18bd9b8ce91d4b95f761f" Oct 30 02:00:07.814871 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:07.814849 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l_odh-ai-gateway-infra(ee325398-ca84-48b8-90b1-105e16fbebd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" podUID="ee325398-ca84-48b8-90b1-105e16fbebd4" Oct 30 02:00:07.816579 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:07.816556 5998 scope.go:122] "RemoveContainer" containerID="20d86d9ae43c284737d85da28db97db489911d269cca09f9ac1ddd8e652adc06" Oct 30 02:00:07.816904 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:07.816872 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5_odh-ai-gateway-infra(ffb61d79-7ce6-46d2-9f0d-712fb8a446a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" podUID="ffb61d79-7ce6-46d2-9f0d-712fb8a446a1" Oct 30 02:00:07.818307 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:07.818287 5998 scope.go:122] "RemoveContainer" containerID="50732e69cb41ce09869a4770965b0f4dbedc30e5f902ef27ec801cbd7297c95c" Oct 30 02:00:07.818583 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:07.818559 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g_odh-ai-gateway-infra(1008ca2d-680d-496b-acec-7ac087a0d48c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" podUID="1008ca2d-680d-496b-acec-7ac087a0d48c" Oct 30 02:00:07.820221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:07.820199 5998 scope.go:122] "RemoveContainer" containerID="7ebb63bae98970cc0bdba9b4f35e306a36cfc7e758f0fd067ae46aa8f7c77e1e" Oct 30 02:00:07.820596 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:07.820474 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888760-rxmvl_odh-ai-gateway-infra(a6132fae-af61-4920-b993-8eb4760609ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" podUID="a6132fae-af61-4920-b993-8eb4760609ff" Oct 30 02:00:07.822413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:07.822382 5998 scope.go:122] "RemoveContainer" containerID="b2719d68be32fd7c8d9d94db29884973a015b1d9048ed0ad0d8788697a55a9ed" Oct 30 02:00:07.822629 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:07.822608 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65_odh-ai-gateway-infra(48178a43-c541-48df-a3cd-5712a41998e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" podUID="48178a43-c541-48df-a3cd-5712a41998e9" Oct 30 02:00:07.867403 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:07.867367 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:08.560517 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:08.559038 5998 patch_prober.go:29] interesting pod/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": read tcp 10.133.0.2:32820->10.133.0.106:15021: read: connection reset by peer" start-of-body= Oct 30 02:00:08.560517 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:08.559655 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" podUID="323ecfe9-7182-4e29-9774-dd3dd728fc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": read tcp 10.133.0.2:32820->10.133.0.106:15021: read: connection reset by peer" Oct 30 02:00:08.837415 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:08.837293 5998 generic.go:356] "Generic (PLEG): container finished" podID="323ecfe9-7182-4e29-9774-dd3dd728fc09" containerID="aa45e4b085d2175b53a045d6483a965484aea40259e90f4dabb71fea592559e1" exitCode=0 Oct 30 02:00:08.837914 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:08.837615 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" event={"ID":"323ecfe9-7182-4e29-9774-dd3dd728fc09","Type":"ContainerDied","Data":"aa45e4b085d2175b53a045d6483a965484aea40259e90f4dabb71fea592559e1"} Oct 30 02:00:08.838862 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:08.838832 5998 scope.go:122] "RemoveContainer" containerID="aa45e4b085d2175b53a045d6483a965484aea40259e90f4dabb71fea592559e1" Oct 30 02:00:08.841186 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:08.841148 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:00:08.841315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:08.841242 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:00:08.841315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:08.841291 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:00:08.912122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:08.911687 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:09.660862 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:09.660824 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:09.848780 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:09.848746 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" event={"ID":"323ecfe9-7182-4e29-9774-dd3dd728fc09","Type":"ContainerStarted","Data":"bcf0f46b790721606aa05544dca4b763434d28b0309cc6d082ba7e37a62c34c3"} Oct 30 02:00:09.864047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:09.864003 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 02:00:09.864247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:09.864198 5998 patch_prober.go:29] interesting pod/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 02:00:09.864306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:09.864271 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" podUID="323ecfe9-7182-4e29-9774-dd3dd728fc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 02:00:10.864404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:10.864366 5998 patch_prober.go:29] interesting pod/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 02:00:10.864802 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:10.864417 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" podUID="323ecfe9-7182-4e29-9774-dd3dd728fc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 02:00:11.105114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:11.100960 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:11.379297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:11.379258 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:11.864644 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:11.864596 5998 patch_prober.go:29] interesting pod/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 02:00:11.865209 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:11.864682 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" podUID="323ecfe9-7182-4e29-9774-dd3dd728fc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 02:00:12.035310 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:12.035272 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq"] Oct 30 02:00:12.045439 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:12.045409 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a6c10e-dlfcq"] Oct 30 02:00:12.542182 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:12.542150 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed6d64f0-4eda-48ba-aa2c-954d0555eb83" path="/var/lib/kubelet/pods/ed6d64f0-4eda-48ba-aa2c-954d0555eb83/volumes" Oct 30 02:00:12.812751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:12.812616 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:12.864871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:12.864834 5998 patch_prober.go:29] interesting pod/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 02:00:12.865305 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:12.864904 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" podUID="323ecfe9-7182-4e29-9774-dd3dd728fc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 02:00:13.864932 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:13.864894 5998 patch_prober.go:29] interesting pod/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 02:00:13.865391 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:13.864950 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" podUID="323ecfe9-7182-4e29-9774-dd3dd728fc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 02:00:14.864916 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:14.864875 5998 patch_prober.go:29] interesting pod/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 02:00:14.865138 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:14.864930 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" podUID="323ecfe9-7182-4e29-9774-dd3dd728fc09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 02:00:14.923923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:14.923714 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:15.531861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.531824 5998 scope.go:122] "RemoveContainer" containerID="3d4281d8cc242d23bd8f9bb0a088fdeab41a8cd5e4e4ccd6b62c9e433c6ab607" Oct 30 02:00:15.742851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.742817 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv"] Oct 30 02:00:15.748970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.748945 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.773066 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.773023 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv"] Oct 30 02:00:15.852347 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.852249 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6ba1d9-openshift-default-dockercfg-67skz\"" Oct 30 02:00:15.870351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.870324 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 02:00:15.870920 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.870888 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 02:00:15.871941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.871923 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 02:00:15.892552 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.892519 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-workload-certs\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.892710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.892658 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daf0043-f037-417f-bf48-7a8f53565395-istiod-ca-cert\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.892769 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.892730 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3daf0043-f037-417f-bf48-7a8f53565395-istio-token\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.892844 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.892823 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-credential-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.892922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.892904 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-workload-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.892978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.892934 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h5nk\" (UniqueName: \"kubernetes.io/projected/3daf0043-f037-417f-bf48-7a8f53565395-kube-api-access-4h5nk\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.892978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.892952 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-istio-envoy\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.893154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.893136 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daf0043-f037-417f-bf48-7a8f53565395-istio-podinfo\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.893190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.893166 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-istio-data\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.994165 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.994122 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daf0043-f037-417f-bf48-7a8f53565395-istio-podinfo\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.994353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.994175 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-istio-data\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.994353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.994273 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-workload-certs\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.994486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.994387 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daf0043-f037-417f-bf48-7a8f53565395-istiod-ca-cert\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.994545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.994489 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3daf0043-f037-417f-bf48-7a8f53565395-istio-token\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.994545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.994523 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-credential-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.994819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.994775 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-istio-data\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.994819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.994811 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-credential-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.995188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.995170 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-workload-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.995294 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.995196 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daf0043-f037-417f-bf48-7a8f53565395-istiod-ca-cert\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.995400 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.995376 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-workload-certs\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.995482 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.995204 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h5nk\" (UniqueName: \"kubernetes.io/projected/3daf0043-f037-417f-bf48-7a8f53565395-kube-api-access-4h5nk\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.995482 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.995442 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-istio-envoy\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.995482 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.995453 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-workload-socket\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:15.999347 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:15.999320 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-istio-envoy\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:16.049111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:16.048561 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daf0043-f037-417f-bf48-7a8f53565395-istio-podinfo\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:16.070044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:16.069958 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h5nk\" (UniqueName: \"kubernetes.io/projected/3daf0043-f037-417f-bf48-7a8f53565395-kube-api-access-4h5nk\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:16.073284 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:16.073249 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3daf0043-f037-417f-bf48-7a8f53565395-istio-token\") pod \"e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:16.365453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:16.365420 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:16.577753 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:16.577690 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:16.700537 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:16.700496 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:16.748893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:16.748852 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv"] Oct 30 02:00:16.761409 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:16.761361 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:00:16.761536 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:16.761450 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:00:16.761536 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:16.761487 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:00:16.824116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:16.819042 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:16.950019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:16.949420 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:16.956472 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:16.956429 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" event={"ID":"3daf0043-f037-417f-bf48-7a8f53565395","Type":"ContainerStarted","Data":"c216991d46a7b1949107e031c52b43465132a3d965e51c21f6db00e3421ce16a"} Oct 30 02:00:16.958985 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:16.958957 5998 generic.go:356] "Generic (PLEG): container finished" podID="af80b61b-f6eb-4e60-abbd-83495acec7d9" containerID="c736fc93d4290fa07bf6e034c4bc883444d3fe8fd815060e00f3ed4973979a97" exitCode=60 Oct 30 02:00:16.959191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:16.959172 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" event={"ID":"af80b61b-f6eb-4e60-abbd-83495acec7d9","Type":"ContainerDied","Data":"c736fc93d4290fa07bf6e034c4bc883444d3fe8fd815060e00f3ed4973979a97"} Oct 30 02:00:16.959343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:16.959330 5998 scope.go:122] "RemoveContainer" containerID="3d4281d8cc242d23bd8f9bb0a088fdeab41a8cd5e4e4ccd6b62c9e433c6ab607" Oct 30 02:00:16.959931 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:16.959916 5998 scope.go:122] "RemoveContainer" containerID="c736fc93d4290fa07bf6e034c4bc883444d3fe8fd815060e00f3ed4973979a97" Oct 30 02:00:16.960525 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:16.960489 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w_odh-ai-gateway-infra(af80b61b-f6eb-4e60-abbd-83495acec7d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" podUID="af80b61b-f6eb-4e60-abbd-83495acec7d9" Oct 30 02:00:17.538112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:17.534875 5998 scope.go:122] "RemoveContainer" containerID="2c1129429b2941151f228cb46424240693839f2c1d8f52e0a86e908b4fcb3b32" Oct 30 02:00:17.765069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:17.765025 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:17.917581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:17.917530 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:18.021115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:18.018703 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" event={"ID":"3daf0043-f037-417f-bf48-7a8f53565395","Type":"ContainerStarted","Data":"880499a7aa319b9434fc6e54076a0df60466d2e0f7f3221653263a8715ef8116"} Oct 30 02:00:18.063257 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:18.063224 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w"] Oct 30 02:00:18.094284 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:18.092355 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" podStartSLOduration=3.092338417 podStartE2EDuration="3.092338417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:18.085305214 +0000 UTC m=+1847.664096295" watchObservedRunningTime="2026-10-30 02:00:18.092338417 +0000 UTC m=+1847.671129499" Oct 30 02:00:18.289521 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:18.289399 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:18.533150 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:18.464812 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:18.533150 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:18.483274 5998 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 02:00:18.533150 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:18.483356 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" podUID="3daf0043-f037-417f-bf48-7a8f53565395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 02:00:18.573123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:18.542731 5998 scope.go:122] "RemoveContainer" containerID="b2719d68be32fd7c8d9d94db29884973a015b1d9048ed0ad0d8788697a55a9ed" Oct 30 02:00:18.573123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:18.550698 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:18.573123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:18.551408 5998 scope.go:122] "RemoveContainer" containerID="ee205f95dc53de6f75096ef8d24135db5a5dc1db844a539ea49d55525f56d5f5" Oct 30 02:00:18.573123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:18.552472 5998 scope.go:122] "RemoveContainer" containerID="7ebb63bae98970cc0bdba9b4f35e306a36cfc7e758f0fd067ae46aa8f7c77e1e" Oct 30 02:00:18.589331 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:18.586607 5998 scope.go:122] "RemoveContainer" containerID="a9cd4e29bb89a8d810cecaf945899ee6a46f8d49d8b18bd9b8ce91d4b95f761f" Oct 30 02:00:18.828390 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:18.807723 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:19.238116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:19.198472 5998 generic.go:356] "Generic (PLEG): container finished" podID="1c0254eb-8332-4172-b1ba-a8235aa882a8" containerID="06e63a89eb661572fbf3f830f1ac4cd5186d322ff5ec14aca6d959c69a20caac" exitCode=60 Oct 30 02:00:19.238116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:19.200007 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" event={"ID":"1c0254eb-8332-4172-b1ba-a8235aa882a8","Type":"ContainerDied","Data":"06e63a89eb661572fbf3f830f1ac4cd5186d322ff5ec14aca6d959c69a20caac"} Oct 30 02:00:19.238116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:19.200147 5998 scope.go:122] "RemoveContainer" containerID="2c1129429b2941151f228cb46424240693839f2c1d8f52e0a86e908b4fcb3b32" Oct 30 02:00:19.238116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:19.200690 5998 scope.go:122] "RemoveContainer" containerID="06e63a89eb661572fbf3f830f1ac4cd5186d322ff5ec14aca6d959c69a20caac" Oct 30 02:00:19.240165 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:19.240127 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x_odh-ai-gateway-infra(1c0254eb-8332-4172-b1ba-a8235aa882a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" podUID="1c0254eb-8332-4172-b1ba-a8235aa882a8" Oct 30 02:00:19.369637 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:19.369598 5998 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 02:00:19.398149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:19.398109 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" podUID="3daf0043-f037-417f-bf48-7a8f53565395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 02:00:19.463388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:19.463350 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" Oct 30 02:00:19.558736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:19.556314 5998 scope.go:122] "RemoveContainer" containerID="20d86d9ae43c284737d85da28db97db489911d269cca09f9ac1ddd8e652adc06" Oct 30 02:00:19.558736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:19.557150 5998 scope.go:122] "RemoveContainer" containerID="50732e69cb41ce09869a4770965b0f4dbedc30e5f902ef27ec801cbd7297c95c" Oct 30 02:00:19.639167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:19.635787 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af80b61b-f6eb-4e60-abbd-83495acec7d9-kube-api-access-2k6ms\" (UniqueName: \"kubernetes.io/projected/af80b61b-f6eb-4e60-abbd-83495acec7d9-kube-api-access-2k6ms\") pod \"af80b61b-f6eb-4e60-abbd-83495acec7d9\" (UID: \"af80b61b-f6eb-4e60-abbd-83495acec7d9\") " Oct 30 02:00:19.639167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:19.635864 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af80b61b-f6eb-4e60-abbd-83495acec7d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af80b61b-f6eb-4e60-abbd-83495acec7d9-maas-api-service-ca\") pod \"af80b61b-f6eb-4e60-abbd-83495acec7d9\" (UID: \"af80b61b-f6eb-4e60-abbd-83495acec7d9\") " Oct 30 02:00:19.639167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:19.638484 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af80b61b-f6eb-4e60-abbd-83495acec7d9-maas-api-service-ca" pod "af80b61b-f6eb-4e60-abbd-83495acec7d9" (UID: "af80b61b-f6eb-4e60-abbd-83495acec7d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:19.697205 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:19.697147 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af80b61b-f6eb-4e60-abbd-83495acec7d9-kube-api-access-2k6ms" pod "af80b61b-f6eb-4e60-abbd-83495acec7d9" (UID: "af80b61b-f6eb-4e60-abbd-83495acec7d9"). InnerVolumeSpecName "kube-api-access-2k6ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:19.775113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:19.739704 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k6ms\" (UniqueName: \"kubernetes.io/projected/af80b61b-f6eb-4e60-abbd-83495acec7d9-kube-api-access-2k6ms\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:19.775113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:19.739741 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af80b61b-f6eb-4e60-abbd-83495acec7d9-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:20.324161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:20.286224 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:20.429193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:20.413052 5998 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 02:00:20.429193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:20.413125 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" podUID="3daf0043-f037-417f-bf48-7a8f53565395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 02:00:20.468484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:20.468448 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x"] Oct 30 02:00:20.497452 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:20.496297 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" event={"ID":"847d96ae-df65-41f4-8663-53fdbd7a4b97","Type":"ContainerStarted","Data":"be72523c9a035254be264b80afa22c87bfaa1374dc7eede4f94486ec2f6e9c1f"} Oct 30 02:00:20.693692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:20.659874 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" event={"ID":"ee325398-ca84-48b8-90b1-105e16fbebd4","Type":"ContainerStarted","Data":"7ac39b2a8e4cc3961eed3725d8a74f53e5773a0b4340426fe85592b14d97133b"} Oct 30 02:00:20.693692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:20.660433 5998 scope.go:122] "RemoveContainer" containerID="7ac39b2a8e4cc3961eed3725d8a74f53e5773a0b4340426fe85592b14d97133b" Oct 30 02:00:20.693692 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:20.660835 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l_odh-ai-gateway-infra(ee325398-ca84-48b8-90b1-105e16fbebd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" podUID="ee325398-ca84-48b8-90b1-105e16fbebd4" Oct 30 02:00:20.712709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:20.708020 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" event={"ID":"af80b61b-f6eb-4e60-abbd-83495acec7d9","Type":"ContainerDied","Data":"948025b5276e69d76050364ef3b0d815ba0a7e218310b68fda5d877541fdce8f"} Oct 30 02:00:20.712709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:20.708120 5998 scope.go:122] "RemoveContainer" containerID="c736fc93d4290fa07bf6e034c4bc883444d3fe8fd815060e00f3ed4973979a97" Oct 30 02:00:20.712709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:20.708221 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w" Oct 30 02:00:20.743780 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:20.743743 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" event={"ID":"a6132fae-af61-4920-b993-8eb4760609ff","Type":"ContainerStarted","Data":"24083e83be49edd222e3fd25d366e2d006764cdb763477e07970d1a90335a396"} Oct 30 02:00:20.757114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:20.744509 5998 scope.go:122] "RemoveContainer" containerID="24083e83be49edd222e3fd25d366e2d006764cdb763477e07970d1a90335a396" Oct 30 02:00:20.757826 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:20.757792 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888760-rxmvl_odh-ai-gateway-infra(a6132fae-af61-4920-b993-8eb4760609ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" podUID="a6132fae-af61-4920-b993-8eb4760609ff" Oct 30 02:00:20.792607 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:20.792568 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" event={"ID":"48178a43-c541-48df-a3cd-5712a41998e9","Type":"ContainerStarted","Data":"34cd9438f262e21ab178f578afca746d60d8d952007066078c155883d4012b40"} Oct 30 02:00:20.793248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:20.793227 5998 scope.go:122] "RemoveContainer" containerID="34cd9438f262e21ab178f578afca746d60d8d952007066078c155883d4012b40" Oct 30 02:00:20.793780 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:20.793753 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65_odh-ai-gateway-infra(48178a43-c541-48df-a3cd-5712a41998e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" podUID="48178a43-c541-48df-a3cd-5712a41998e9" Oct 30 02:00:21.069370 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.069297 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w"] Oct 30 02:00:21.073150 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.073118 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888760-bzw4w"] Oct 30 02:00:21.190409 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.190029 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" Oct 30 02:00:21.268585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.268084 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c0254eb-8332-4172-b1ba-a8235aa882a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c0254eb-8332-4172-b1ba-a8235aa882a8-maas-api-service-ca\") pod \"1c0254eb-8332-4172-b1ba-a8235aa882a8\" (UID: \"1c0254eb-8332-4172-b1ba-a8235aa882a8\") " Oct 30 02:00:21.268585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.268389 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0254eb-8332-4172-b1ba-a8235aa882a8-kube-api-access-h7w87\" (UniqueName: \"kubernetes.io/projected/1c0254eb-8332-4172-b1ba-a8235aa882a8-kube-api-access-h7w87\") pod \"1c0254eb-8332-4172-b1ba-a8235aa882a8\" (UID: \"1c0254eb-8332-4172-b1ba-a8235aa882a8\") " Oct 30 02:00:21.270052 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.268902 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0254eb-8332-4172-b1ba-a8235aa882a8-maas-api-service-ca" pod "1c0254eb-8332-4172-b1ba-a8235aa882a8" (UID: "1c0254eb-8332-4172-b1ba-a8235aa882a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:21.272469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.272442 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0254eb-8332-4172-b1ba-a8235aa882a8-kube-api-access-h7w87" pod "1c0254eb-8332-4172-b1ba-a8235aa882a8" (UID: "1c0254eb-8332-4172-b1ba-a8235aa882a8"). InnerVolumeSpecName "kube-api-access-h7w87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:21.366162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.366079 5998 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 02:00:21.366545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.366162 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" podUID="3daf0043-f037-417f-bf48-7a8f53565395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 02:00:21.370327 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.370231 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c0254eb-8332-4172-b1ba-a8235aa882a8-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.370327 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.370256 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7w87\" (UniqueName: \"kubernetes.io/projected/1c0254eb-8332-4172-b1ba-a8235aa882a8-kube-api-access-h7w87\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.576312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.576229 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:21.593009 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.592977 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5"] Oct 30 02:00:21.783563 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.783526 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l"] Oct 30 02:00:21.823276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.823240 5998 generic.go:356] "Generic (PLEG): container finished" podID="847d96ae-df65-41f4-8663-53fdbd7a4b97" containerID="be72523c9a035254be264b80afa22c87bfaa1374dc7eede4f94486ec2f6e9c1f" exitCode=60 Oct 30 02:00:21.824808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.824775 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" event={"ID":"847d96ae-df65-41f4-8663-53fdbd7a4b97","Type":"ContainerDied","Data":"be72523c9a035254be264b80afa22c87bfaa1374dc7eede4f94486ec2f6e9c1f"} Oct 30 02:00:21.825034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.825015 5998 scope.go:122] "RemoveContainer" containerID="ee205f95dc53de6f75096ef8d24135db5a5dc1db844a539ea49d55525f56d5f5" Oct 30 02:00:21.846061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.846019 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" event={"ID":"1c0254eb-8332-4172-b1ba-a8235aa882a8","Type":"ContainerDied","Data":"cb0b9422003fe76ebadf783f00128fd8fc8ceff72abaddd2af9e1da734f1c2f1"} Oct 30 02:00:21.846444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.846409 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x" Oct 30 02:00:21.849988 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.849955 5998 generic.go:356] "Generic (PLEG): container finished" podID="ee325398-ca84-48b8-90b1-105e16fbebd4" containerID="7ac39b2a8e4cc3961eed3725d8a74f53e5773a0b4340426fe85592b14d97133b" exitCode=60 Oct 30 02:00:21.850102 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.850046 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" event={"ID":"ee325398-ca84-48b8-90b1-105e16fbebd4","Type":"ContainerDied","Data":"7ac39b2a8e4cc3961eed3725d8a74f53e5773a0b4340426fe85592b14d97133b"} Oct 30 02:00:21.855315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.853871 5998 generic.go:356] "Generic (PLEG): container finished" podID="ffb61d79-7ce6-46d2-9f0d-712fb8a446a1" containerID="6ed26695742ecf4788321bede6b33d58d2599b8f2b0aa637f265b0261ed9fe8e" exitCode=60 Oct 30 02:00:21.855315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.853965 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" event={"ID":"ffb61d79-7ce6-46d2-9f0d-712fb8a446a1","Type":"ContainerDied","Data":"6ed26695742ecf4788321bede6b33d58d2599b8f2b0aa637f265b0261ed9fe8e"} Oct 30 02:00:21.855315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.854440 5998 scope.go:122] "RemoveContainer" containerID="6ed26695742ecf4788321bede6b33d58d2599b8f2b0aa637f265b0261ed9fe8e" Oct 30 02:00:21.855315 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:21.854826 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5_odh-ai-gateway-infra(ffb61d79-7ce6-46d2-9f0d-712fb8a446a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" podUID="ffb61d79-7ce6-46d2-9f0d-712fb8a446a1" Oct 30 02:00:21.859237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.859215 5998 generic.go:356] "Generic (PLEG): container finished" podID="1008ca2d-680d-496b-acec-7ac087a0d48c" containerID="675cd89776a40d3ed65e724e000351e270ec2a1dd1b5d2aaa7e7ac17b9c2b8b0" exitCode=60 Oct 30 02:00:21.859442 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.859424 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" event={"ID":"1008ca2d-680d-496b-acec-7ac087a0d48c","Type":"ContainerDied","Data":"675cd89776a40d3ed65e724e000351e270ec2a1dd1b5d2aaa7e7ac17b9c2b8b0"} Oct 30 02:00:21.860840 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.860813 5998 scope.go:122] "RemoveContainer" containerID="675cd89776a40d3ed65e724e000351e270ec2a1dd1b5d2aaa7e7ac17b9c2b8b0" Oct 30 02:00:21.861658 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:21.861636 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g_odh-ai-gateway-infra(1008ca2d-680d-496b-acec-7ac087a0d48c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" podUID="1008ca2d-680d-496b-acec-7ac087a0d48c" Oct 30 02:00:21.862756 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.862735 5998 generic.go:356] "Generic (PLEG): container finished" podID="a6132fae-af61-4920-b993-8eb4760609ff" containerID="24083e83be49edd222e3fd25d366e2d006764cdb763477e07970d1a90335a396" exitCode=60 Oct 30 02:00:21.862894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.862818 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" event={"ID":"a6132fae-af61-4920-b993-8eb4760609ff","Type":"ContainerDied","Data":"24083e83be49edd222e3fd25d366e2d006764cdb763477e07970d1a90335a396"} Oct 30 02:00:21.863341 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.863328 5998 scope.go:122] "RemoveContainer" containerID="24083e83be49edd222e3fd25d366e2d006764cdb763477e07970d1a90335a396" Oct 30 02:00:21.863825 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:21.863686 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888760-rxmvl_odh-ai-gateway-infra(a6132fae-af61-4920-b993-8eb4760609ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" podUID="a6132fae-af61-4920-b993-8eb4760609ff" Oct 30 02:00:21.864382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.864365 5998 scope.go:122] "RemoveContainer" containerID="06e63a89eb661572fbf3f830f1ac4cd5186d322ff5ec14aca6d959c69a20caac" Oct 30 02:00:21.866044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.866021 5998 generic.go:356] "Generic (PLEG): container finished" podID="48178a43-c541-48df-a3cd-5712a41998e9" containerID="34cd9438f262e21ab178f578afca746d60d8d952007066078c155883d4012b40" exitCode=60 Oct 30 02:00:21.866173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.866141 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" event={"ID":"48178a43-c541-48df-a3cd-5712a41998e9","Type":"ContainerDied","Data":"34cd9438f262e21ab178f578afca746d60d8d952007066078c155883d4012b40"} Oct 30 02:00:21.866504 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.866485 5998 scope.go:122] "RemoveContainer" containerID="34cd9438f262e21ab178f578afca746d60d8d952007066078c155883d4012b40" Oct 30 02:00:21.866874 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:21.866853 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65_odh-ai-gateway-infra(48178a43-c541-48df-a3cd-5712a41998e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" podUID="48178a43-c541-48df-a3cd-5712a41998e9" Oct 30 02:00:21.893180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.892465 5998 scope.go:122] "RemoveContainer" containerID="a9cd4e29bb89a8d810cecaf945899ee6a46f8d49d8b18bd9b8ce91d4b95f761f" Oct 30 02:00:21.967040 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:21.967015 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65"] Oct 30 02:00:22.050134 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.039468 5998 scope.go:122] "RemoveContainer" containerID="20d86d9ae43c284737d85da28db97db489911d269cca09f9ac1ddd8e652adc06" Oct 30 02:00:22.080664 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.078379 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:22.080664 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.078572 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl"] Oct 30 02:00:22.221230 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.221153 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x"] Oct 30 02:00:22.285261 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.257728 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:22.378116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.371529 5998 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 02:00:22.378116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.371587 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" podUID="3daf0043-f037-417f-bf48-7a8f53565395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 02:00:22.386075 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.383267 5998 scope.go:122] "RemoveContainer" containerID="50732e69cb41ce09869a4770965b0f4dbedc30e5f902ef27ec801cbd7297c95c" Oct 30 02:00:22.444133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.439907 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-954773-29888760-lhh8x"] Oct 30 02:00:22.529328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.519584 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:22.693814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.693777 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c0254eb-8332-4172-b1ba-a8235aa882a8" path="/var/lib/kubelet/pods/1c0254eb-8332-4172-b1ba-a8235aa882a8/volumes" Oct 30 02:00:22.694567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.694543 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af80b61b-f6eb-4e60-abbd-83495acec7d9" path="/var/lib/kubelet/pods/af80b61b-f6eb-4e60-abbd-83495acec7d9/volumes" Oct 30 02:00:22.769298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.766617 5998 scope.go:122] "RemoveContainer" containerID="7ebb63bae98970cc0bdba9b4f35e306a36cfc7e758f0fd067ae46aa8f7c77e1e" Oct 30 02:00:22.784251 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.784222 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" Oct 30 02:00:22.850890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.848750 5998 scope.go:122] "RemoveContainer" containerID="b2719d68be32fd7c8d9d94db29884973a015b1d9048ed0ad0d8788697a55a9ed" Oct 30 02:00:22.863997 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.863632 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" Oct 30 02:00:22.885508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.885336 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" Oct 30 02:00:22.885508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.885370 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5" event={"ID":"847d96ae-df65-41f4-8663-53fdbd7a4b97","Type":"ContainerDied","Data":"819f6c425c7bc20989f5c535068fa26e9d7392fd7ccd7bd55d9c729f1b1349fd"} Oct 30 02:00:22.885508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.885432 5998 scope.go:122] "RemoveContainer" containerID="be72523c9a035254be264b80afa22c87bfaa1374dc7eede4f94486ec2f6e9c1f" Oct 30 02:00:22.889920 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.889882 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" event={"ID":"ee325398-ca84-48b8-90b1-105e16fbebd4","Type":"ContainerDied","Data":"533f53c6039cff8c194fdec79253267cee8d6a883f9a4c3d26a9249c8e8df816"} Oct 30 02:00:22.890038 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.889894 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l" Oct 30 02:00:22.918189 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.918164 5998 scope.go:122] "RemoveContainer" containerID="7ac39b2a8e4cc3961eed3725d8a74f53e5773a0b4340426fe85592b14d97133b" Oct 30 02:00:22.988551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.975615 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/847d96ae-df65-41f4-8663-53fdbd7a4b97-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/847d96ae-df65-41f4-8663-53fdbd7a4b97-maas-api-service-ca\") pod \"847d96ae-df65-41f4-8663-53fdbd7a4b97\" (UID: \"847d96ae-df65-41f4-8663-53fdbd7a4b97\") " Oct 30 02:00:22.988551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.975739 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee325398-ca84-48b8-90b1-105e16fbebd4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee325398-ca84-48b8-90b1-105e16fbebd4-maas-api-service-ca\") pod \"ee325398-ca84-48b8-90b1-105e16fbebd4\" (UID: \"ee325398-ca84-48b8-90b1-105e16fbebd4\") " Oct 30 02:00:22.988551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.975825 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee325398-ca84-48b8-90b1-105e16fbebd4-kube-api-access-ghhb8\" (UniqueName: \"kubernetes.io/projected/ee325398-ca84-48b8-90b1-105e16fbebd4-kube-api-access-ghhb8\") pod \"ee325398-ca84-48b8-90b1-105e16fbebd4\" (UID: \"ee325398-ca84-48b8-90b1-105e16fbebd4\") " Oct 30 02:00:22.988551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.975967 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/847d96ae-df65-41f4-8663-53fdbd7a4b97-kube-api-access-4bq54\" (UniqueName: \"kubernetes.io/projected/847d96ae-df65-41f4-8663-53fdbd7a4b97-kube-api-access-4bq54\") pod \"847d96ae-df65-41f4-8663-53fdbd7a4b97\" (UID: \"847d96ae-df65-41f4-8663-53fdbd7a4b97\") " Oct 30 02:00:22.988551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.977817 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee325398-ca84-48b8-90b1-105e16fbebd4-maas-api-service-ca" pod "ee325398-ca84-48b8-90b1-105e16fbebd4" (UID: "ee325398-ca84-48b8-90b1-105e16fbebd4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:22.988551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.978207 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/847d96ae-df65-41f4-8663-53fdbd7a4b97-maas-api-service-ca" pod "847d96ae-df65-41f4-8663-53fdbd7a4b97" (UID: "847d96ae-df65-41f4-8663-53fdbd7a4b97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:22.988551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.979600 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847d96ae-df65-41f4-8663-53fdbd7a4b97-kube-api-access-4bq54" pod "847d96ae-df65-41f4-8663-53fdbd7a4b97" (UID: "847d96ae-df65-41f4-8663-53fdbd7a4b97"). InnerVolumeSpecName "kube-api-access-4bq54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:22.988551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.987855 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:22.992302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:22.992227 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee325398-ca84-48b8-90b1-105e16fbebd4-kube-api-access-ghhb8" pod "ee325398-ca84-48b8-90b1-105e16fbebd4" (UID: "ee325398-ca84-48b8-90b1-105e16fbebd4"). InnerVolumeSpecName "kube-api-access-ghhb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:23.177151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.149141 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/847d96ae-df65-41f4-8663-53fdbd7a4b97-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.177151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.149175 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee325398-ca84-48b8-90b1-105e16fbebd4-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.177151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.149192 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghhb8\" (UniqueName: \"kubernetes.io/projected/ee325398-ca84-48b8-90b1-105e16fbebd4-kube-api-access-ghhb8\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.177151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.149205 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bq54\" (UniqueName: \"kubernetes.io/projected/847d96ae-df65-41f4-8663-53fdbd7a4b97-kube-api-access-4bq54\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.177151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.160515 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5"] Oct 30 02:00:23.177544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.177392 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch"] Oct 30 02:00:23.177544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.177445 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12e41f89-prsch"] Oct 30 02:00:23.217711 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.217619 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:23.293493 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.289865 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g"] Oct 30 02:00:23.391208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.391175 5998 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 02:00:23.391772 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.391737 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" podUID="3daf0043-f037-417f-bf48-7a8f53565395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 02:00:23.465840 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.464382 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l"] Oct 30 02:00:23.475598 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.475564 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-f6f4f0-29888760-bb77l"] Oct 30 02:00:23.512503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.512410 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" Oct 30 02:00:23.518185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.518153 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5"] Oct 30 02:00:23.538898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.538865 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-6ba1d9-29888760-zvsp5"] Oct 30 02:00:23.664136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.663992 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" Oct 30 02:00:23.764308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.753766 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6132fae-af61-4920-b993-8eb4760609ff-kube-api-access-w4sjq\" (UniqueName: \"kubernetes.io/projected/a6132fae-af61-4920-b993-8eb4760609ff-kube-api-access-w4sjq\") pod \"a6132fae-af61-4920-b993-8eb4760609ff\" (UID: \"a6132fae-af61-4920-b993-8eb4760609ff\") " Oct 30 02:00:23.764308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.753870 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48178a43-c541-48df-a3cd-5712a41998e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48178a43-c541-48df-a3cd-5712a41998e9-maas-api-service-ca\") pod \"48178a43-c541-48df-a3cd-5712a41998e9\" (UID: \"48178a43-c541-48df-a3cd-5712a41998e9\") " Oct 30 02:00:23.764308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.753967 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48178a43-c541-48df-a3cd-5712a41998e9-kube-api-access-pwzzv\" (UniqueName: \"kubernetes.io/projected/48178a43-c541-48df-a3cd-5712a41998e9-kube-api-access-pwzzv\") pod \"48178a43-c541-48df-a3cd-5712a41998e9\" (UID: \"48178a43-c541-48df-a3cd-5712a41998e9\") " Oct 30 02:00:23.764308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.754125 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6132fae-af61-4920-b993-8eb4760609ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6132fae-af61-4920-b993-8eb4760609ff-maas-api-service-ca\") pod \"a6132fae-af61-4920-b993-8eb4760609ff\" (UID: \"a6132fae-af61-4920-b993-8eb4760609ff\") " Oct 30 02:00:23.764308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.758518 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48178a43-c541-48df-a3cd-5712a41998e9-maas-api-service-ca" pod "48178a43-c541-48df-a3cd-5712a41998e9" (UID: "48178a43-c541-48df-a3cd-5712a41998e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:23.766723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.765619 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6132fae-af61-4920-b993-8eb4760609ff-maas-api-service-ca" pod "a6132fae-af61-4920-b993-8eb4760609ff" (UID: "a6132fae-af61-4920-b993-8eb4760609ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:23.768773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.768366 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48178a43-c541-48df-a3cd-5712a41998e9-kube-api-access-pwzzv" pod "48178a43-c541-48df-a3cd-5712a41998e9" (UID: "48178a43-c541-48df-a3cd-5712a41998e9"). InnerVolumeSpecName "kube-api-access-pwzzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:23.768773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.768428 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6132fae-af61-4920-b993-8eb4760609ff-kube-api-access-w4sjq" pod "a6132fae-af61-4920-b993-8eb4760609ff" (UID: "a6132fae-af61-4920-b993-8eb4760609ff"). InnerVolumeSpecName "kube-api-access-w4sjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:23.860024 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.858719 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwzzv\" (UniqueName: \"kubernetes.io/projected/48178a43-c541-48df-a3cd-5712a41998e9-kube-api-access-pwzzv\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.860024 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.858758 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6132fae-af61-4920-b993-8eb4760609ff-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.860024 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.858876 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4sjq\" (UniqueName: \"kubernetes.io/projected/a6132fae-af61-4920-b993-8eb4760609ff-kube-api-access-w4sjq\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.860024 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:23.858893 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48178a43-c541-48df-a3cd-5712a41998e9-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:24.043195 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.030532 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" Oct 30 02:00:24.095139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.091330 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" event={"ID":"ffb61d79-7ce6-46d2-9f0d-712fb8a446a1","Type":"ContainerDied","Data":"44be394eeea54e18bceb4c15fb69cc08e34a6895d384b4128081e51c9801c248"} Oct 30 02:00:24.095139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.091404 5998 scope.go:122] "RemoveContainer" containerID="6ed26695742ecf4788321bede6b33d58d2599b8f2b0aa637f265b0261ed9fe8e" Oct 30 02:00:24.095139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.091534 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5" Oct 30 02:00:24.095139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.093513 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" Oct 30 02:00:24.095559 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.095528 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" event={"ID":"1008ca2d-680d-496b-acec-7ac087a0d48c","Type":"ContainerDied","Data":"1d36c2e99fe49b9347c255c7742bee10506c7fb5ae95e1603f9ede3a3acae39c"} Oct 30 02:00:24.095699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.095676 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb61d79-7ce6-46d2-9f0d-712fb8a446a1-kube-api-access-tb8tp\" (UniqueName: \"kubernetes.io/projected/ffb61d79-7ce6-46d2-9f0d-712fb8a446a1-kube-api-access-tb8tp\") pod \"ffb61d79-7ce6-46d2-9f0d-712fb8a446a1\" (UID: \"ffb61d79-7ce6-46d2-9f0d-712fb8a446a1\") " Oct 30 02:00:24.095846 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.095832 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb61d79-7ce6-46d2-9f0d-712fb8a446a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb61d79-7ce6-46d2-9f0d-712fb8a446a1-maas-api-service-ca\") pod \"ffb61d79-7ce6-46d2-9f0d-712fb8a446a1\" (UID: \"ffb61d79-7ce6-46d2-9f0d-712fb8a446a1\") " Oct 30 02:00:24.097188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.097159 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb61d79-7ce6-46d2-9f0d-712fb8a446a1-maas-api-service-ca" pod "ffb61d79-7ce6-46d2-9f0d-712fb8a446a1" (UID: "ffb61d79-7ce6-46d2-9f0d-712fb8a446a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:24.099900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.099870 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" event={"ID":"a6132fae-af61-4920-b993-8eb4760609ff","Type":"ContainerDied","Data":"90138c9c56ed876807b21d2e6b1ccb1e92565c788a3d991d614eed5bb289fede"} Oct 30 02:00:24.100024 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.099956 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl" Oct 30 02:00:24.100506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.100138 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb61d79-7ce6-46d2-9f0d-712fb8a446a1-kube-api-access-tb8tp" pod "ffb61d79-7ce6-46d2-9f0d-712fb8a446a1" (UID: "ffb61d79-7ce6-46d2-9f0d-712fb8a446a1"). InnerVolumeSpecName "kube-api-access-tb8tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:24.105106 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.102073 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" event={"ID":"48178a43-c541-48df-a3cd-5712a41998e9","Type":"ContainerDied","Data":"efb73f1a7bd597e8a59b3cd7123d880807268f69d420a8c33554da23572da778"} Oct 30 02:00:24.105106 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.102189 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65" Oct 30 02:00:24.169158 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.162467 5998 scope.go:122] "RemoveContainer" containerID="675cd89776a40d3ed65e724e000351e270ec2a1dd1b5d2aaa7e7ac17b9c2b8b0" Oct 30 02:00:24.188167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.186331 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65"] Oct 30 02:00:24.199646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.199608 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1008ca2d-680d-496b-acec-7ac087a0d48c-kube-api-access-gv9w6\" (UniqueName: \"kubernetes.io/projected/1008ca2d-680d-496b-acec-7ac087a0d48c-kube-api-access-gv9w6\") pod \"1008ca2d-680d-496b-acec-7ac087a0d48c\" (UID: \"1008ca2d-680d-496b-acec-7ac087a0d48c\") " Oct 30 02:00:24.199807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.199725 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1008ca2d-680d-496b-acec-7ac087a0d48c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1008ca2d-680d-496b-acec-7ac087a0d48c-maas-api-service-ca\") pod \"1008ca2d-680d-496b-acec-7ac087a0d48c\" (UID: \"1008ca2d-680d-496b-acec-7ac087a0d48c\") " Oct 30 02:00:24.244116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.204684 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb8tp\" (UniqueName: \"kubernetes.io/projected/ffb61d79-7ce6-46d2-9f0d-712fb8a446a1-kube-api-access-tb8tp\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:24.244116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.204718 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffb61d79-7ce6-46d2-9f0d-712fb8a446a1-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:24.244116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.207302 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1008ca2d-680d-496b-acec-7ac087a0d48c-maas-api-service-ca" pod "1008ca2d-680d-496b-acec-7ac087a0d48c" (UID: "1008ca2d-680d-496b-acec-7ac087a0d48c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:24.262536 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.262461 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:24.277126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.277037 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c40a89-29888760-68v65"] Oct 30 02:00:24.279764 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.278642 5998 scope.go:122] "RemoveContainer" containerID="24083e83be49edd222e3fd25d366e2d006764cdb763477e07970d1a90335a396" Oct 30 02:00:24.295362 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.295293 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1008ca2d-680d-496b-acec-7ac087a0d48c-kube-api-access-gv9w6" pod "1008ca2d-680d-496b-acec-7ac087a0d48c" (UID: "1008ca2d-680d-496b-acec-7ac087a0d48c"). InnerVolumeSpecName "kube-api-access-gv9w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:24.299858 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.299833 5998 scope.go:122] "RemoveContainer" containerID="34cd9438f262e21ab178f578afca746d60d8d952007066078c155883d4012b40" Oct 30 02:00:24.325272 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.325238 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv9w6\" (UniqueName: \"kubernetes.io/projected/1008ca2d-680d-496b-acec-7ac087a0d48c-kube-api-access-gv9w6\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:24.335424 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.335383 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1008ca2d-680d-496b-acec-7ac087a0d48c-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:24.345496 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.345428 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl"] Oct 30 02:00:24.372673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.372608 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888760-rxmvl"] Oct 30 02:00:24.373821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.373154 5998 patch_prober.go:29] interesting pod/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 02:00:24.373821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.373201 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" podUID="3daf0043-f037-417f-bf48-7a8f53565395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 02:00:24.377824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.377788 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp"] Oct 30 02:00:24.378264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.378211 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp" podUID="9901c539-93e6-407d-b99c-b7cd2ad64356" containerName="payload-pre-processing" containerID="cri-o://c2ab70ea125c238febfa054866f8d33f9e25142e0a8e19cd7edef46a42b75463" gracePeriod=30 Oct 30 02:00:24.485033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.475598 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:24.581260 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.581219 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48178a43-c541-48df-a3cd-5712a41998e9" path="/var/lib/kubelet/pods/48178a43-c541-48df-a3cd-5712a41998e9/volumes" Oct 30 02:00:24.581769 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.581751 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="847d96ae-df65-41f4-8663-53fdbd7a4b97" path="/var/lib/kubelet/pods/847d96ae-df65-41f4-8663-53fdbd7a4b97/volumes" Oct 30 02:00:24.582570 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.582546 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6132fae-af61-4920-b993-8eb4760609ff" path="/var/lib/kubelet/pods/a6132fae-af61-4920-b993-8eb4760609ff/volumes" Oct 30 02:00:24.583559 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.583532 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee325398-ca84-48b8-90b1-105e16fbebd4" path="/var/lib/kubelet/pods/ee325398-ca84-48b8-90b1-105e16fbebd4/volumes" Oct 30 02:00:24.583989 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.583969 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f95dafc6-6159-4e69-9ab1-1090d5b244d1" path="/var/lib/kubelet/pods/f95dafc6-6159-4e69-9ab1-1090d5b244d1/volumes" Oct 30 02:00:24.584624 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.584607 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5"] Oct 30 02:00:24.584804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.584789 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-5c90f5-29888760-hbct5"] Oct 30 02:00:24.674267 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.665515 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:24.798946 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.798917 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:24.826584 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.826153 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp" Oct 30 02:00:24.917132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.913958 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:24.950546 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.949773 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9901c539-93e6-407d-b99c-b7cd2ad64356-kube-api-access-bdxf6\" (UniqueName: \"kubernetes.io/projected/9901c539-93e6-407d-b99c-b7cd2ad64356-kube-api-access-bdxf6\") pod \"9901c539-93e6-407d-b99c-b7cd2ad64356\" (UID: \"9901c539-93e6-407d-b99c-b7cd2ad64356\") " Oct 30 02:00:24.950546 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.949852 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9901c539-93e6-407d-b99c-b7cd2ad64356-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9901c539-93e6-407d-b99c-b7cd2ad64356-plugins-config-volume\") pod \"9901c539-93e6-407d-b99c-b7cd2ad64356\" (UID: \"9901c539-93e6-407d-b99c-b7cd2ad64356\") " Oct 30 02:00:24.951011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.950977 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9901c539-93e6-407d-b99c-b7cd2ad64356-plugins-config-volume" pod "9901c539-93e6-407d-b99c-b7cd2ad64356" (UID: "9901c539-93e6-407d-b99c-b7cd2ad64356"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:24.952390 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:24.952347 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9901c539-93e6-407d-b99c-b7cd2ad64356-kube-api-access-bdxf6" pod "9901c539-93e6-407d-b99c-b7cd2ad64356" (UID: "9901c539-93e6-407d-b99c-b7cd2ad64356"). InnerVolumeSpecName "kube-api-access-bdxf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:25.056113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:25.055798 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdxf6\" (UniqueName: \"kubernetes.io/projected/9901c539-93e6-407d-b99c-b7cd2ad64356-kube-api-access-bdxf6\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:25.056113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:25.055850 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9901c539-93e6-407d-b99c-b7cd2ad64356-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:25.294834 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:25.294733 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:25.298139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:25.297077 5998 generic.go:356] "Generic (PLEG): container finished" podID="9901c539-93e6-407d-b99c-b7cd2ad64356" containerID="c2ab70ea125c238febfa054866f8d33f9e25142e0a8e19cd7edef46a42b75463" exitCode=0 Oct 30 02:00:25.298139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:25.297318 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp" event={"ID":"9901c539-93e6-407d-b99c-b7cd2ad64356","Type":"ContainerDied","Data":"c2ab70ea125c238febfa054866f8d33f9e25142e0a8e19cd7edef46a42b75463"} Oct 30 02:00:25.298139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:25.297376 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp" event={"ID":"9901c539-93e6-407d-b99c-b7cd2ad64356","Type":"ContainerDied","Data":"57d8b328ae9fc27ea6b14dda13ac2693b6f2969bb893d49857e130d227d6e103"} Oct 30 02:00:25.298139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:25.297411 5998 scope.go:122] "RemoveContainer" containerID="c2ab70ea125c238febfa054866f8d33f9e25142e0a8e19cd7edef46a42b75463" Oct 30 02:00:25.298139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:25.297534 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp" Oct 30 02:00:25.321366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:25.319700 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g" Oct 30 02:00:25.350800 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:25.350761 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:00:25.358344 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:25.358312 5998 scope.go:122] "RemoveContainer" containerID="c2ab70ea125c238febfa054866f8d33f9e25142e0a8e19cd7edef46a42b75463" Oct 30 02:00:25.359208 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:25.359175 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ab70ea125c238febfa054866f8d33f9e25142e0a8e19cd7edef46a42b75463\": container with ID starting with c2ab70ea125c238febfa054866f8d33f9e25142e0a8e19cd7edef46a42b75463 not found: ID does not exist" containerID="c2ab70ea125c238febfa054866f8d33f9e25142e0a8e19cd7edef46a42b75463" Oct 30 02:00:25.359348 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:25.359218 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ab70ea125c238febfa054866f8d33f9e25142e0a8e19cd7edef46a42b75463"} err="failed to get container status \"c2ab70ea125c238febfa054866f8d33f9e25142e0a8e19cd7edef46a42b75463\": rpc error: code = NotFound desc = could not find container \"c2ab70ea125c238febfa054866f8d33f9e25142e0a8e19cd7edef46a42b75463\": container with ID starting with c2ab70ea125c238febfa054866f8d33f9e25142e0a8e19cd7edef46a42b75463 not found: ID does not exist" Oct 30 02:00:25.373891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:25.373863 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:25.374513 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:25.374483 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:25.375213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:25.375196 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:00:25.382907 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:25.382879 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g"] Oct 30 02:00:25.387743 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:25.387719 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888760-5mp8g"] Oct 30 02:00:25.421422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:25.421387 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp"] Oct 30 02:00:25.425694 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:25.425664 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd098e-69b494c675-59gcp"] Oct 30 02:00:26.307476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:26.307415 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:26.452738 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:26.452709 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:26.542672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:26.542639 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1008ca2d-680d-496b-acec-7ac087a0d48c" path="/var/lib/kubelet/pods/1008ca2d-680d-496b-acec-7ac087a0d48c/volumes" Oct 30 02:00:26.543147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:26.543133 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9901c539-93e6-407d-b99c-b7cd2ad64356" path="/var/lib/kubelet/pods/9901c539-93e6-407d-b99c-b7cd2ad64356/volumes" Oct 30 02:00:26.543519 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:26.543506 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffb61d79-7ce6-46d2-9f0d-712fb8a446a1" path="/var/lib/kubelet/pods/ffb61d79-7ce6-46d2-9f0d-712fb8a446a1/volumes" Oct 30 02:00:27.396112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:27.392557 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:27.515342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:27.515308 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:27.714568 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:27.714537 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:28.130869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:28.130785 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:28.243632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:28.243550 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:28.527171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:28.526737 5998 generic.go:356] "Generic (PLEG): container finished" podID="f8e03b95-821f-456c-a435-831d0bfa52f0" containerID="9ca51b1842e73bbb0337773e16edb2615575f589a2c25aaaeaaa959531c4aa23" exitCode=0 Oct 30 02:00:28.527171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:28.526840 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" event={"ID":"f8e03b95-821f-456c-a435-831d0bfa52f0","Type":"ContainerDied","Data":"9ca51b1842e73bbb0337773e16edb2615575f589a2c25aaaeaaa959531c4aa23"} Oct 30 02:00:28.527714 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:28.527380 5998 scope.go:122] "RemoveContainer" containerID="9ca51b1842e73bbb0337773e16edb2615575f589a2c25aaaeaaa959531c4aa23" Oct 30 02:00:28.573258 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:28.560507 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:00:28.573258 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:28.560629 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:00:28.573258 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:28.560684 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:00:28.692324 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:28.692216 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:28.751859 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:28.751828 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8"] Oct 30 02:00:28.752412 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:28.752383 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" podUID="5c72e866-870f-4f28-8e32-eb303f84488d" containerName="istio-proxy" containerID="cri-o://e13191c869b4297cc7184685ff51cef1d7a02c8c25a7dabdb89a892b66699d2e" gracePeriod=30 Oct 30 02:00:29.539126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:29.539071 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" event={"ID":"f8e03b95-821f-456c-a435-831d0bfa52f0","Type":"ContainerStarted","Data":"c56a2860f48966a3356b49c3b13c68743ff64be9de71f58be01bd78e7e4ff846"} Oct 30 02:00:29.676438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:29.676406 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 02:00:29.677309 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:29.677275 5998 patch_prober.go:29] interesting pod/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 02:00:29.677456 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:29.677330 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" podUID="f8e03b95-821f-456c-a435-831d0bfa52f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 02:00:30.365506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:30.365449 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:30.676592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:30.676561 5998 patch_prober.go:29] interesting pod/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 02:00:30.676989 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:30.676625 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" podUID="f8e03b95-821f-456c-a435-831d0bfa52f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 02:00:31.488457 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:31.488421 5998 scope.go:122] "RemoveContainer" containerID="253118fb5e4b060d257828eedcbe9dcb8fafc81bce932fde79414c069de98dc5" Oct 30 02:00:31.498936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:31.498881 5998 scope.go:122] "RemoveContainer" containerID="27edfbb90be82e7753a3ee20167cf11843ff72ecb768d4a93ec914fdc8e5ee12" Oct 30 02:00:31.536310 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:31.536285 5998 scope.go:122] "RemoveContainer" containerID="c4c947996d97740bc2b8836c7bfcfa81b671002c8c24186cec20750d9748ded1" Oct 30 02:00:31.548056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:31.548030 5998 scope.go:122] "RemoveContainer" containerID="b45b6d19b00ab4d3ae3964b4d61f8b237e7d91b43caf4079048f1641006d8da1" Oct 30 02:00:31.559198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:31.559172 5998 scope.go:122] "RemoveContainer" containerID="046b87f3bf629f27b063686cfbd200299fe32c810dad7c40b05839fcb75eb540" Oct 30 02:00:31.570028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:31.570010 5998 scope.go:122] "RemoveContainer" containerID="45fd57e9751848a3a193bd3afaf1263d706cdf57d0fa9966aa9e7be2f12110cb" Oct 30 02:00:31.677245 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:31.677214 5998 patch_prober.go:29] interesting pod/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 02:00:31.677661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:31.677271 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" podUID="f8e03b95-821f-456c-a435-831d0bfa52f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 02:00:32.539351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:32.539271 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:32.645438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:32.645399 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:32.676717 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:32.676686 5998 patch_prober.go:29] interesting pod/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 02:00:32.676914 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:32.676744 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" podUID="f8e03b95-821f-456c-a435-831d0bfa52f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 02:00:32.763145 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:32.763025 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:33.677823 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:33.677529 5998 patch_prober.go:29] interesting pod/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 02:00:33.678313 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:33.678276 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" podUID="f8e03b95-821f-456c-a435-831d0bfa52f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 02:00:34.037807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.037754 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 02:00:34.152682 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.152649 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 02:00:34.176247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.176210 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:34.233499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.225947 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c72e866-870f-4f28-8e32-eb303f84488d-kube-api-access-5jnth\" (UniqueName: \"kubernetes.io/projected/5c72e866-870f-4f28-8e32-eb303f84488d-kube-api-access-5jnth\") pod \"5c72e866-870f-4f28-8e32-eb303f84488d\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " Oct 30 02:00:34.233499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.226005 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-istio-envoy\") pod \"5c72e866-870f-4f28-8e32-eb303f84488d\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " Oct 30 02:00:34.233499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.226064 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-credential-socket\") pod \"5c72e866-870f-4f28-8e32-eb303f84488d\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " Oct 30 02:00:34.233499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.226184 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-workload-certs\") pod \"5c72e866-870f-4f28-8e32-eb303f84488d\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " Oct 30 02:00:34.233499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.226232 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c72e866-870f-4f28-8e32-eb303f84488d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c72e866-870f-4f28-8e32-eb303f84488d-istiod-ca-cert\") pod \"5c72e866-870f-4f28-8e32-eb303f84488d\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " Oct 30 02:00:34.233499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.226274 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c72e866-870f-4f28-8e32-eb303f84488d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c72e866-870f-4f28-8e32-eb303f84488d-istio-podinfo\") pod \"5c72e866-870f-4f28-8e32-eb303f84488d\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " Oct 30 02:00:34.233499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.226393 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c72e866-870f-4f28-8e32-eb303f84488d-istio-token\" (UniqueName: \"kubernetes.io/projected/5c72e866-870f-4f28-8e32-eb303f84488d-istio-token\") pod \"5c72e866-870f-4f28-8e32-eb303f84488d\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " Oct 30 02:00:34.233499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.226501 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-workload-socket\") pod \"5c72e866-870f-4f28-8e32-eb303f84488d\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " Oct 30 02:00:34.233499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.226530 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-istio-data\") pod \"5c72e866-870f-4f28-8e32-eb303f84488d\" (UID: \"5c72e866-870f-4f28-8e32-eb303f84488d\") " Oct 30 02:00:34.233499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.232380 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-workload-certs" pod "5c72e866-870f-4f28-8e32-eb303f84488d" (UID: "5c72e866-870f-4f28-8e32-eb303f84488d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:34.245657 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.245612 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c72e866-870f-4f28-8e32-eb303f84488d-istiod-ca-cert" pod "5c72e866-870f-4f28-8e32-eb303f84488d" (UID: "5c72e866-870f-4f28-8e32-eb303f84488d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:34.247014 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.246835 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-workload-socket" pod "5c72e866-870f-4f28-8e32-eb303f84488d" (UID: "5c72e866-870f-4f28-8e32-eb303f84488d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:34.247829 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.247203 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-credential-socket" pod "5c72e866-870f-4f28-8e32-eb303f84488d" (UID: "5c72e866-870f-4f28-8e32-eb303f84488d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:34.252192 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.252033 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c72e866-870f-4f28-8e32-eb303f84488d-kube-api-access-5jnth" pod "5c72e866-870f-4f28-8e32-eb303f84488d" (UID: "5c72e866-870f-4f28-8e32-eb303f84488d"). InnerVolumeSpecName "kube-api-access-5jnth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:34.260177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.255504 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-istio-data" pod "5c72e866-870f-4f28-8e32-eb303f84488d" (UID: "5c72e866-870f-4f28-8e32-eb303f84488d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:34.260177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.258792 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-istio-envoy" pod "5c72e866-870f-4f28-8e32-eb303f84488d" (UID: "5c72e866-870f-4f28-8e32-eb303f84488d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:00:34.260177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.259730 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c72e866-870f-4f28-8e32-eb303f84488d-istio-token" pod "5c72e866-870f-4f28-8e32-eb303f84488d" (UID: "5c72e866-870f-4f28-8e32-eb303f84488d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:34.263137 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.262249 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c72e866-870f-4f28-8e32-eb303f84488d-istio-podinfo" pod "5c72e866-870f-4f28-8e32-eb303f84488d" (UID: "5c72e866-870f-4f28-8e32-eb303f84488d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:00:34.333114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.331383 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:34.333114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.331424 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:34.333114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.331439 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jnth\" (UniqueName: \"kubernetes.io/projected/5c72e866-870f-4f28-8e32-eb303f84488d-kube-api-access-5jnth\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:34.333114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.331455 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:34.333114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.331469 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:34.333114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.331482 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c72e866-870f-4f28-8e32-eb303f84488d-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:34.333114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.331496 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c72e866-870f-4f28-8e32-eb303f84488d-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:34.333114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.331508 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c72e866-870f-4f28-8e32-eb303f84488d-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:34.333114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.331525 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c72e866-870f-4f28-8e32-eb303f84488d-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:00:34.333114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.332997 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:34.585489 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.585455 5998 generic.go:356] "Generic (PLEG): container finished" podID="5c72e866-870f-4f28-8e32-eb303f84488d" containerID="e13191c869b4297cc7184685ff51cef1d7a02c8c25a7dabdb89a892b66699d2e" exitCode=0 Oct 30 02:00:34.585686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.585510 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" Oct 30 02:00:34.585686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.585543 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" event={"ID":"5c72e866-870f-4f28-8e32-eb303f84488d","Type":"ContainerDied","Data":"e13191c869b4297cc7184685ff51cef1d7a02c8c25a7dabdb89a892b66699d2e"} Oct 30 02:00:34.585686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.585618 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8" event={"ID":"5c72e866-870f-4f28-8e32-eb303f84488d","Type":"ContainerDied","Data":"228e15568aef22b401a8146be21d33f6d0190f50bdc5b83cadc81a02ce739a8b"} Oct 30 02:00:34.585686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.585650 5998 scope.go:122] "RemoveContainer" containerID="e13191c869b4297cc7184685ff51cef1d7a02c8c25a7dabdb89a892b66699d2e" Oct 30 02:00:34.597237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.597217 5998 scope.go:122] "RemoveContainer" containerID="5cc57826cfaa636799c44bbfe5d9b689489f4f106e10254fa7718143388ce11b" Oct 30 02:00:34.646701 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.646663 5998 scope.go:122] "RemoveContainer" containerID="e13191c869b4297cc7184685ff51cef1d7a02c8c25a7dabdb89a892b66699d2e" Oct 30 02:00:34.647137 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:34.647110 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13191c869b4297cc7184685ff51cef1d7a02c8c25a7dabdb89a892b66699d2e\": container with ID starting with e13191c869b4297cc7184685ff51cef1d7a02c8c25a7dabdb89a892b66699d2e not found: ID does not exist" containerID="e13191c869b4297cc7184685ff51cef1d7a02c8c25a7dabdb89a892b66699d2e" Oct 30 02:00:34.647216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.647154 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13191c869b4297cc7184685ff51cef1d7a02c8c25a7dabdb89a892b66699d2e"} err="failed to get container status \"e13191c869b4297cc7184685ff51cef1d7a02c8c25a7dabdb89a892b66699d2e\": rpc error: code = NotFound desc = could not find container \"e13191c869b4297cc7184685ff51cef1d7a02c8c25a7dabdb89a892b66699d2e\": container with ID starting with e13191c869b4297cc7184685ff51cef1d7a02c8c25a7dabdb89a892b66699d2e not found: ID does not exist" Oct 30 02:00:34.647216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.647179 5998 scope.go:122] "RemoveContainer" containerID="5cc57826cfaa636799c44bbfe5d9b689489f4f106e10254fa7718143388ce11b" Oct 30 02:00:34.647575 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:34.647548 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc57826cfaa636799c44bbfe5d9b689489f4f106e10254fa7718143388ce11b\": container with ID starting with 5cc57826cfaa636799c44bbfe5d9b689489f4f106e10254fa7718143388ce11b not found: ID does not exist" containerID="5cc57826cfaa636799c44bbfe5d9b689489f4f106e10254fa7718143388ce11b" Oct 30 02:00:34.647650 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.647583 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc57826cfaa636799c44bbfe5d9b689489f4f106e10254fa7718143388ce11b"} err="failed to get container status \"5cc57826cfaa636799c44bbfe5d9b689489f4f106e10254fa7718143388ce11b\": rpc error: code = NotFound desc = could not find container \"5cc57826cfaa636799c44bbfe5d9b689489f4f106e10254fa7718143388ce11b\": container with ID starting with 5cc57826cfaa636799c44bbfe5d9b689489f4f106e10254fa7718143388ce11b not found: ID does not exist" Oct 30 02:00:34.655002 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.654975 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8"] Oct 30 02:00:34.665550 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.665518 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bd098e-openshift-default-8b546f85f-wlkf8"] Oct 30 02:00:34.676633 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.676604 5998 patch_prober.go:29] interesting pod/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 02:00:34.676744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:34.676663 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" podUID="f8e03b95-821f-456c-a435-831d0bfa52f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 02:00:35.037964 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:35.037927 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc"] Oct 30 02:00:35.044058 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:35.044019 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2bbe2c6-62hnc"] Oct 30 02:00:35.350558 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:35.350462 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:35.681385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:35.681353 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 02:00:35.682574 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:35.682551 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 02:00:35.999562 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:35.999467 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:36.543183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:36.543150 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c20a06-9416-4e1d-91d4-7eb88d1f6d98" path="/var/lib/kubelet/pods/59c20a06-9416-4e1d-91d4-7eb88d1f6d98/volumes" Oct 30 02:00:36.543819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:36.543799 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c72e866-870f-4f28-8e32-eb303f84488d" path="/var/lib/kubelet/pods/5c72e866-870f-4f28-8e32-eb303f84488d/volumes" Oct 30 02:00:38.082359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:38.082324 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:38.100575 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:38.100541 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:00:39.381747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:39.381715 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:39.579576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:39.579530 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:40.804075 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:40.804038 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:42.041970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:42.041932 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:42.239794 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:42.238903 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:42.442839 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:42.437362 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:43.569344 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:43.569307 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:45.374721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:45.374683 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:47.047741 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:47.047698 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:48.705625 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:48.705585 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:50.339646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:50.339584 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:50.475141 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:50.470519 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:52.354406 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:52.354376 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:54.086110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:54.082230 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:54.197760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:54.197721 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:55.222074 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:00:55.222031 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:00:55.777248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:55.777210 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:56.327438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:56.327406 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:57.461929 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:57.461865 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:59.251491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:59.251442 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:59.426359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:59.426271 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:59.536424 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:59.536342 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:00:59.798160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:00:59.792429 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:00.216456 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:00.216388 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:02.123647 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:02.123606 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:03.898392 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:03.898356 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:04.033522 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:04.033473 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5"] Oct 30 02:01:04.044029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:04.043982 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-970f9605-rhhg5"] Oct 30 02:01:04.540012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:04.539973 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb7ac19d-73e6-4435-b070-16d6db455913" path="/var/lib/kubelet/pods/eb7ac19d-73e6-4435-b070-16d6db455913/volumes" Oct 30 02:01:06.271944 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:06.271901 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:06.304121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:06.294369 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:01:06.418567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:06.418530 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:06.877325 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:06.877283 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:08.936304 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:08.936255 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:09.072214 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:09.072180 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:09.199141 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:09.199051 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:09.338169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:09.338127 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:09.659582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:09.659497 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:10.057147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:10.057107 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67"] Oct 30 02:01:10.057651 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:10.057383 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" podUID="46590fa3-d39f-4df5-8c5b-96520faa7625" containerName="istio-proxy" containerID="cri-o://c0ccbf7540ec62ffa995f28751b050f8eacaef1531ccca6f18ffbbdc1eda561a" gracePeriod=30 Oct 30 02:01:10.071966 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:10.071925 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc"] Oct 30 02:01:10.072461 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:10.072420 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" podUID="19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" containerName="istio-proxy" containerID="cri-o://540f538545780d1b5e7253eddbc9122328154546448d0e3393ecfe5ad14b82e2" gracePeriod=30 Oct 30 02:01:10.093955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:10.093929 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:11.658316 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:11.658279 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:11.884072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:11.884003 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:12.180294 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:12.180257 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:12.665296 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:12.661670 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:13.621889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:13.621856 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:14.319740 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:14.319697 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:14.428947 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:14.428906 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:15.089435 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.089398 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn"] Oct 30 02:01:15.089673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.089644 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" podUID="a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" containerName="istio-proxy" containerID="cri-o://aef65e1106b7ffb44a2e604aa32084565c216447430dc8fd93258f188c4b23a9" gracePeriod=30 Oct 30 02:01:15.105222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.105192 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w"] Oct 30 02:01:15.110878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.105421 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" podUID="3605446f-2664-4b96-9a9f-e13aab64faee" containerName="istio-proxy" containerID="cri-o://883b0ca245b89825ab2c2b7bccd313bc9c11f9633acbaafebbaccfb62607a603" gracePeriod=30 Oct 30 02:01:15.120394 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.120362 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm"] Oct 30 02:01:15.120864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.120832 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" podUID="0b6f67a0-2dae-400a-abd7-7c65fbdac225" containerName="istio-proxy" containerID="cri-o://3d85368bb06bb388c8f248f640ccf045c6d5eed581fad5fbf75b276628ea0ba0" gracePeriod=30 Oct 30 02:01:15.126627 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.126453 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t"] Oct 30 02:01:15.126787 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.126688 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" podUID="7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" containerName="istio-proxy" containerID="cri-o://1f20376a1a3e8e4701c80f9584c3e1b1080554df0fa72ddaa99af2a76d5bf4dc" gracePeriod=30 Oct 30 02:01:15.423478 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.423449 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 02:01:15.456425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.456402 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 02:01:15.504490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.504456 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-kube-api-access-svj7g\" (UniqueName: \"kubernetes.io/projected/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-kube-api-access-svj7g\") pod \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " Oct 30 02:01:15.504724 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.504705 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-workload-certs\") pod \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " Oct 30 02:01:15.504866 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.504852 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-credential-socket\") pod \"46590fa3-d39f-4df5-8c5b-96520faa7625\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " Oct 30 02:01:15.504995 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.504982 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-envoy\") pod \"46590fa3-d39f-4df5-8c5b-96520faa7625\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " Oct 30 02:01:15.505128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.505113 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-podinfo\") pod \"46590fa3-d39f-4df5-8c5b-96520faa7625\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " Oct 30 02:01:15.505340 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.505324 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46590fa3-d39f-4df5-8c5b-96520faa7625-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46590fa3-d39f-4df5-8c5b-96520faa7625-istiod-ca-cert\") pod \"46590fa3-d39f-4df5-8c5b-96520faa7625\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " Oct 30 02:01:15.505458 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.505446 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-credential-socket\") pod \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " Oct 30 02:01:15.505567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.505552 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-data\") pod \"46590fa3-d39f-4df5-8c5b-96520faa7625\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " Oct 30 02:01:15.506169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.505145 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-workload-certs" pod "19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" (UID: "19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.506169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.505322 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-credential-socket" pod "46590fa3-d39f-4df5-8c5b-96520faa7625" (UID: "46590fa3-d39f-4df5-8c5b-96520faa7625"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.506169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.505893 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-data" pod "46590fa3-d39f-4df5-8c5b-96520faa7625" (UID: "46590fa3-d39f-4df5-8c5b-96520faa7625"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.506169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.505945 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-credential-socket" pod "19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" (UID: "19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.506169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.505992 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46590fa3-d39f-4df5-8c5b-96520faa7625-istiod-ca-cert" pod "46590fa3-d39f-4df5-8c5b-96520faa7625" (UID: "46590fa3-d39f-4df5-8c5b-96520faa7625"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:15.506572 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.506552 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-token\" (UniqueName: \"kubernetes.io/projected/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-token\") pod \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " Oct 30 02:01:15.506991 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.506974 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-workload-socket\") pod \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " Oct 30 02:01:15.507498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.507393 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-workload-socket" pod "19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" (UID: "19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.507661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.507644 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istiod-ca-cert\") pod \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " Oct 30 02:01:15.508131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.508034 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istiod-ca-cert" pod "19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" (UID: "19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:15.508279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.508261 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46590fa3-d39f-4df5-8c5b-96520faa7625-kube-api-access-gkqgq\" (UniqueName: \"kubernetes.io/projected/46590fa3-d39f-4df5-8c5b-96520faa7625-kube-api-access-gkqgq\") pod \"46590fa3-d39f-4df5-8c5b-96520faa7625\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " Oct 30 02:01:15.508663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.508647 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-data\") pod \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " Oct 30 02:01:15.509128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.509018 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-data" pod "19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" (UID: "19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.509277 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.509261 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-token\" (UniqueName: \"kubernetes.io/projected/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-token\") pod \"46590fa3-d39f-4df5-8c5b-96520faa7625\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " Oct 30 02:01:15.509691 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.509673 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-podinfo\") pod \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " Oct 30 02:01:15.509832 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.509817 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-workload-socket\") pod \"46590fa3-d39f-4df5-8c5b-96520faa7625\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " Oct 30 02:01:15.509969 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.509956 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-envoy\") pod \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\" (UID: \"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8\") " Oct 30 02:01:15.510073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.510060 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-workload-certs\") pod \"46590fa3-d39f-4df5-8c5b-96520faa7625\" (UID: \"46590fa3-d39f-4df5-8c5b-96520faa7625\") " Oct 30 02:01:15.510939 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.510917 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.511062 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.511050 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.511194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.511182 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.511288 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.511277 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.511423 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.511410 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.511518 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.511505 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46590fa3-d39f-4df5-8c5b-96520faa7625-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.511633 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.511622 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.511753 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.511742 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.511864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.510964 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-workload-certs" pod "46590fa3-d39f-4df5-8c5b-96520faa7625" (UID: "46590fa3-d39f-4df5-8c5b-96520faa7625"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.512071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.511017 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-workload-socket" pod "46590fa3-d39f-4df5-8c5b-96520faa7625" (UID: "46590fa3-d39f-4df5-8c5b-96520faa7625"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.512196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.511997 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-token" pod "19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" (UID: "19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:15.521688 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.521645 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46590fa3-d39f-4df5-8c5b-96520faa7625-kube-api-access-gkqgq" pod "46590fa3-d39f-4df5-8c5b-96520faa7625" (UID: "46590fa3-d39f-4df5-8c5b-96520faa7625"). InnerVolumeSpecName "kube-api-access-gkqgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:15.521815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.521684 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-kube-api-access-svj7g" pod "19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" (UID: "19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8"). InnerVolumeSpecName "kube-api-access-svj7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:15.521815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.521698 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-podinfo" pod "46590fa3-d39f-4df5-8c5b-96520faa7625" (UID: "46590fa3-d39f-4df5-8c5b-96520faa7625"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:01:15.521815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.521726 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-token" pod "46590fa3-d39f-4df5-8c5b-96520faa7625" (UID: "46590fa3-d39f-4df5-8c5b-96520faa7625"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:15.521815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.521755 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-envoy" pod "46590fa3-d39f-4df5-8c5b-96520faa7625" (UID: "46590fa3-d39f-4df5-8c5b-96520faa7625"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.522427 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.522392 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-podinfo" pod "19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" (UID: "19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:01:15.522932 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.522910 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-envoy" pod "19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" (UID: "19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:15.613366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.613279 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkqgq\" (UniqueName: \"kubernetes.io/projected/46590fa3-d39f-4df5-8c5b-96520faa7625-kube-api-access-gkqgq\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.613366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.613314 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.613366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.613324 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.613366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.613332 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.613366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.613342 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.613366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.613353 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.613366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.613364 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svj7g\" (UniqueName: \"kubernetes.io/projected/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-kube-api-access-svj7g\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.613723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.613373 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.613723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.613382 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46590fa3-d39f-4df5-8c5b-96520faa7625-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.613723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.613389 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:15.658029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.657994 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:15.868273 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.868173 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:15.985799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.985764 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:15.998859 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.998826 5998 generic.go:356] "Generic (PLEG): container finished" podID="19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" containerID="540f538545780d1b5e7253eddbc9122328154546448d0e3393ecfe5ad14b82e2" exitCode=0 Oct 30 02:01:15.999022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.998888 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" Oct 30 02:01:15.999022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.998905 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" event={"ID":"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8","Type":"ContainerDied","Data":"540f538545780d1b5e7253eddbc9122328154546448d0e3393ecfe5ad14b82e2"} Oct 30 02:01:15.999022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.998987 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc" event={"ID":"19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8","Type":"ContainerDied","Data":"f1cf8b27f5dbe4fd8abec51fad0ad2b375bdef621cccf80f1450fe9de07a26d0"} Oct 30 02:01:15.999022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:15.999015 5998 scope.go:122] "RemoveContainer" containerID="540f538545780d1b5e7253eddbc9122328154546448d0e3393ecfe5ad14b82e2" Oct 30 02:01:16.000525 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:16.000501 5998 generic.go:356] "Generic (PLEG): container finished" podID="46590fa3-d39f-4df5-8c5b-96520faa7625" containerID="c0ccbf7540ec62ffa995f28751b050f8eacaef1531ccca6f18ffbbdc1eda561a" exitCode=0 Oct 30 02:01:16.000646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:16.000576 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" Oct 30 02:01:16.000646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:16.000586 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" event={"ID":"46590fa3-d39f-4df5-8c5b-96520faa7625","Type":"ContainerDied","Data":"c0ccbf7540ec62ffa995f28751b050f8eacaef1531ccca6f18ffbbdc1eda561a"} Oct 30 02:01:16.000765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:16.000697 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67" event={"ID":"46590fa3-d39f-4df5-8c5b-96520faa7625","Type":"ContainerDied","Data":"26b9f0d4e80b9f15e6b291722d6391c899fb1fb37c6a91f16e79cfbecc44ecac"} Oct 30 02:01:16.010751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:16.010709 5998 scope.go:122] "RemoveContainer" containerID="540f538545780d1b5e7253eddbc9122328154546448d0e3393ecfe5ad14b82e2" Oct 30 02:01:16.012051 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:01:16.011332 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"540f538545780d1b5e7253eddbc9122328154546448d0e3393ecfe5ad14b82e2\": container with ID starting with 540f538545780d1b5e7253eddbc9122328154546448d0e3393ecfe5ad14b82e2 not found: ID does not exist" containerID="540f538545780d1b5e7253eddbc9122328154546448d0e3393ecfe5ad14b82e2" Oct 30 02:01:16.012051 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:16.011370 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"540f538545780d1b5e7253eddbc9122328154546448d0e3393ecfe5ad14b82e2"} err="failed to get container status \"540f538545780d1b5e7253eddbc9122328154546448d0e3393ecfe5ad14b82e2\": rpc error: code = NotFound desc = could not find container \"540f538545780d1b5e7253eddbc9122328154546448d0e3393ecfe5ad14b82e2\": container with ID starting with 540f538545780d1b5e7253eddbc9122328154546448d0e3393ecfe5ad14b82e2 not found: ID does not exist" Oct 30 02:01:16.012051 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:16.011390 5998 scope.go:122] "RemoveContainer" containerID="c0ccbf7540ec62ffa995f28751b050f8eacaef1531ccca6f18ffbbdc1eda561a" Oct 30 02:01:16.025310 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:16.025288 5998 scope.go:122] "RemoveContainer" containerID="c0ccbf7540ec62ffa995f28751b050f8eacaef1531ccca6f18ffbbdc1eda561a" Oct 30 02:01:16.025604 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:01:16.025583 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ccbf7540ec62ffa995f28751b050f8eacaef1531ccca6f18ffbbdc1eda561a\": container with ID starting with c0ccbf7540ec62ffa995f28751b050f8eacaef1531ccca6f18ffbbdc1eda561a not found: ID does not exist" containerID="c0ccbf7540ec62ffa995f28751b050f8eacaef1531ccca6f18ffbbdc1eda561a" Oct 30 02:01:16.025700 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:16.025613 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ccbf7540ec62ffa995f28751b050f8eacaef1531ccca6f18ffbbdc1eda561a"} err="failed to get container status \"c0ccbf7540ec62ffa995f28751b050f8eacaef1531ccca6f18ffbbdc1eda561a\": rpc error: code = NotFound desc = could not find container \"c0ccbf7540ec62ffa995f28751b050f8eacaef1531ccca6f18ffbbdc1eda561a\": container with ID starting with c0ccbf7540ec62ffa995f28751b050f8eacaef1531ccca6f18ffbbdc1eda561a not found: ID does not exist" Oct 30 02:01:16.066844 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:16.066811 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc"] Oct 30 02:01:16.077527 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:16.077491 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-sthhc"] Oct 30 02:01:16.108616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:16.108584 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67"] Oct 30 02:01:16.112426 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:16.112388 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-86dd79-openshift-default-b5cd9fb85-k7v67"] Oct 30 02:01:16.543236 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:16.543197 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" path="/var/lib/kubelet/pods/19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8/volumes" Oct 30 02:01:16.543987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:16.543966 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46590fa3-d39f-4df5-8c5b-96520faa7625" path="/var/lib/kubelet/pods/46590fa3-d39f-4df5-8c5b-96520faa7625/volumes" Oct 30 02:01:16.774389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:16.774356 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:17.643462 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:17.643425 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:17.807449 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:17.807415 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:19.577276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:19.577241 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:20.455920 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.455889 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 02:01:20.491301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.491230 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 02:01:20.562654 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.562626 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 02:01:20.566850 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.566818 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 02:01:20.597526 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.597488 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3605446f-2664-4b96-9a9f-e13aab64faee-istiod-ca-cert" pod "3605446f-2664-4b96-9a9f-e13aab64faee" (UID: "3605446f-2664-4b96-9a9f-e13aab64faee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:20.598062 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.597565 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3605446f-2664-4b96-9a9f-e13aab64faee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3605446f-2664-4b96-9a9f-e13aab64faee-istiod-ca-cert\") pod \"3605446f-2664-4b96-9a9f-e13aab64faee\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " Oct 30 02:01:20.598062 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.597656 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-workload-certs\") pod \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " Oct 30 02:01:20.598062 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.597958 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-workload-certs" pod "a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" (UID: "a9c105b9-9c67-48ab-821c-b16c9f9ef5e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.599254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.599228 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3605446f-2664-4b96-9a9f-e13aab64faee-istio-token\" (UniqueName: \"kubernetes.io/projected/3605446f-2664-4b96-9a9f-e13aab64faee-istio-token\") pod \"3605446f-2664-4b96-9a9f-e13aab64faee\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " Oct 30 02:01:20.599646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.599625 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istiod-ca-cert\") pod \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " Oct 30 02:01:20.600063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.600018 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istiod-ca-cert" pod "a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" (UID: "a9c105b9-9c67-48ab-821c-b16c9f9ef5e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:20.600414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.600397 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-kube-api-access-qpjzt\" (UniqueName: \"kubernetes.io/projected/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-kube-api-access-qpjzt\") pod \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " Oct 30 02:01:20.600895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.600878 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-credential-socket\") pod \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " Oct 30 02:01:20.601644 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.601339 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-credential-socket" pod "a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" (UID: "a9c105b9-9c67-48ab-821c-b16c9f9ef5e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.602109 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.602061 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-workload-socket" pod "3605446f-2664-4b96-9a9f-e13aab64faee" (UID: "3605446f-2664-4b96-9a9f-e13aab64faee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.602339 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.602320 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3605446f-2664-4b96-9a9f-e13aab64faee-istio-token" pod "3605446f-2664-4b96-9a9f-e13aab64faee" (UID: "3605446f-2664-4b96-9a9f-e13aab64faee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:20.602444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.601824 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-workload-socket\") pod \"3605446f-2664-4b96-9a9f-e13aab64faee\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " Oct 30 02:01:20.602444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.602394 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-data\") pod \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " Oct 30 02:01:20.602558 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.602489 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-credential-socket\") pod \"3605446f-2664-4b96-9a9f-e13aab64faee\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " Oct 30 02:01:20.602558 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.602525 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-workload-certs\") pod \"3605446f-2664-4b96-9a9f-e13aab64faee\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " Oct 30 02:01:20.602694 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.602650 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-data" pod "a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" (UID: "a9c105b9-9c67-48ab-821c-b16c9f9ef5e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.602855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.602831 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-credential-socket" pod "3605446f-2664-4b96-9a9f-e13aab64faee" (UID: "3605446f-2664-4b96-9a9f-e13aab64faee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.602925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.602864 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-workload-certs" pod "3605446f-2664-4b96-9a9f-e13aab64faee" (UID: "3605446f-2664-4b96-9a9f-e13aab64faee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.602925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.602877 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-istio-envoy\") pod \"3605446f-2664-4b96-9a9f-e13aab64faee\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " Oct 30 02:01:20.602925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.602903 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3605446f-2664-4b96-9a9f-e13aab64faee-kube-api-access-hxkwp\" (UniqueName: \"kubernetes.io/projected/3605446f-2664-4b96-9a9f-e13aab64faee-kube-api-access-hxkwp\") pod \"3605446f-2664-4b96-9a9f-e13aab64faee\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " Oct 30 02:01:20.603112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.602929 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3605446f-2664-4b96-9a9f-e13aab64faee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3605446f-2664-4b96-9a9f-e13aab64faee-istio-podinfo\") pod \"3605446f-2664-4b96-9a9f-e13aab64faee\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " Oct 30 02:01:20.603112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.602965 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-workload-socket\") pod \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " Oct 30 02:01:20.603112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.602993 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-podinfo\") pod \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " Oct 30 02:01:20.603283 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.603117 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-token\" (UniqueName: \"kubernetes.io/projected/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-token\") pod \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " Oct 30 02:01:20.603283 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.603149 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-istio-data\") pod \"3605446f-2664-4b96-9a9f-e13aab64faee\" (UID: \"3605446f-2664-4b96-9a9f-e13aab64faee\") " Oct 30 02:01:20.603283 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.603254 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-envoy\") pod \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\" (UID: \"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5\") " Oct 30 02:01:20.603917 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.603652 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-workload-socket" pod "a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" (UID: "a9c105b9-9c67-48ab-821c-b16c9f9ef5e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.604366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.604331 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-istio-data" pod "3605446f-2664-4b96-9a9f-e13aab64faee" (UID: "3605446f-2664-4b96-9a9f-e13aab64faee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.604801 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.604768 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-kube-api-access-qpjzt" pod "a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" (UID: "a9c105b9-9c67-48ab-821c-b16c9f9ef5e5"). InnerVolumeSpecName "kube-api-access-qpjzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:20.605130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.605103 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3605446f-2664-4b96-9a9f-e13aab64faee-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.605130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.605129 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.605280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.605142 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3605446f-2664-4b96-9a9f-e13aab64faee-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.605280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.605153 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.605280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.605165 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpjzt\" (UniqueName: \"kubernetes.io/projected/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-kube-api-access-qpjzt\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.605280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.605180 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.605280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.605194 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.605280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.605207 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.605280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.605219 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.605280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.605231 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.605280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.605249 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.605280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.605261 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.605661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.605325 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-istio-envoy" pod "3605446f-2664-4b96-9a9f-e13aab64faee" (UID: "3605446f-2664-4b96-9a9f-e13aab64faee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.607749 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.607721 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-token" pod "a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" (UID: "a9c105b9-9c67-48ab-821c-b16c9f9ef5e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:20.607901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.607885 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-envoy" pod "a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" (UID: "a9c105b9-9c67-48ab-821c-b16c9f9ef5e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.608580 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.608551 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3605446f-2664-4b96-9a9f-e13aab64faee-istio-podinfo" pod "3605446f-2664-4b96-9a9f-e13aab64faee" (UID: "3605446f-2664-4b96-9a9f-e13aab64faee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:01:20.608930 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.608906 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-podinfo" pod "a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" (UID: "a9c105b9-9c67-48ab-821c-b16c9f9ef5e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:01:20.609073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.609054 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3605446f-2664-4b96-9a9f-e13aab64faee-kube-api-access-hxkwp" pod "3605446f-2664-4b96-9a9f-e13aab64faee" (UID: "3605446f-2664-4b96-9a9f-e13aab64faee"). InnerVolumeSpecName "kube-api-access-hxkwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:20.706254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.706153 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-credential-socket\") pod \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " Oct 30 02:01:20.706254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.706211 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-kube-api-access-4vpf9\" (UniqueName: \"kubernetes.io/projected/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-kube-api-access-4vpf9\") pod \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " Oct 30 02:01:20.706254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.706240 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6f67a0-2dae-400a-abd7-7c65fbdac225-kube-api-access-cwlkn\" (UniqueName: \"kubernetes.io/projected/0b6f67a0-2dae-400a-abd7-7c65fbdac225-kube-api-access-cwlkn\") pod \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " Oct 30 02:01:20.706581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.706274 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-podinfo\") pod \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " Oct 30 02:01:20.706581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.706317 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istiod-ca-cert\") pod \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " Oct 30 02:01:20.706581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.706359 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-workload-certs\") pod \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " Oct 30 02:01:20.706581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.706401 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-workload-socket\") pod \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " Oct 30 02:01:20.706581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.706450 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-token\" (UniqueName: \"kubernetes.io/projected/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-token\") pod \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " Oct 30 02:01:20.706581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.706561 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-credential-socket" pod "0b6f67a0-2dae-400a-abd7-7c65fbdac225" (UID: "0b6f67a0-2dae-400a-abd7-7c65fbdac225"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.706898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.706605 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istiod-ca-cert\") pod \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " Oct 30 02:01:20.706898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.706637 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-token\" (UniqueName: \"kubernetes.io/projected/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-token\") pod \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " Oct 30 02:01:20.706898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.706752 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-credential-socket\") pod \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " Oct 30 02:01:20.706898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.706794 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-podinfo\") pod \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " Oct 30 02:01:20.706898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.706853 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-envoy\") pod \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " Oct 30 02:01:20.706898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.706885 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istiod-ca-cert" pod "0b6f67a0-2dae-400a-abd7-7c65fbdac225" (UID: "0b6f67a0-2dae-400a-abd7-7c65fbdac225"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:20.707250 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.706953 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-data\") pod \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " Oct 30 02:01:20.707250 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.706982 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-workload-socket\") pod \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " Oct 30 02:01:20.707250 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.707017 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-data\") pod \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " Oct 30 02:01:20.707250 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.707060 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-workload-certs\") pod \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\" (UID: \"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f\") " Oct 30 02:01:20.707250 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.707133 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-envoy\") pod \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\" (UID: \"0b6f67a0-2dae-400a-abd7-7c65fbdac225\") " Oct 30 02:01:20.707511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.707290 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istiod-ca-cert" pod "7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" (UID: "7bbcaa62-d63a-4bf4-ae72-8b091238cc2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:20.707858 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.707685 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-workload-socket" pod "7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" (UID: "7bbcaa62-d63a-4bf4-ae72-8b091238cc2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.707986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.707858 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.707986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.707881 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.707986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.707894 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.707986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.707908 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.707986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.707922 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3605446f-2664-4b96-9a9f-e13aab64faee-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.707986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.707937 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxkwp\" (UniqueName: \"kubernetes.io/projected/3605446f-2664-4b96-9a9f-e13aab64faee-kube-api-access-hxkwp\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.707986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.707951 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3605446f-2664-4b96-9a9f-e13aab64faee-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.707986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.707965 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.707986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.707978 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.707986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.707990 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.708526 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.708280 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-data" pod "0b6f67a0-2dae-400a-abd7-7c65fbdac225" (UID: "0b6f67a0-2dae-400a-abd7-7c65fbdac225"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.708526 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.708447 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-workload-certs" pod "7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" (UID: "7bbcaa62-d63a-4bf4-ae72-8b091238cc2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.709367 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.708689 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-data" pod "7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" (UID: "7bbcaa62-d63a-4bf4-ae72-8b091238cc2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.709367 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.708926 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-workload-certs" pod "0b6f67a0-2dae-400a-abd7-7c65fbdac225" (UID: "0b6f67a0-2dae-400a-abd7-7c65fbdac225"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.709367 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.709117 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-credential-socket" pod "7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" (UID: "7bbcaa62-d63a-4bf4-ae72-8b091238cc2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.709875 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.709855 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-workload-socket" pod "0b6f67a0-2dae-400a-abd7-7c65fbdac225" (UID: "0b6f67a0-2dae-400a-abd7-7c65fbdac225"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.711667 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.711547 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6f67a0-2dae-400a-abd7-7c65fbdac225-kube-api-access-cwlkn" pod "0b6f67a0-2dae-400a-abd7-7c65fbdac225" (UID: "0b6f67a0-2dae-400a-abd7-7c65fbdac225"). InnerVolumeSpecName "kube-api-access-cwlkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:20.711667 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.711619 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-token" pod "0b6f67a0-2dae-400a-abd7-7c65fbdac225" (UID: "0b6f67a0-2dae-400a-abd7-7c65fbdac225"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:20.711914 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.711867 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-envoy" pod "7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" (UID: "7bbcaa62-d63a-4bf4-ae72-8b091238cc2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.711914 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.711883 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-podinfo" pod "7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" (UID: "7bbcaa62-d63a-4bf4-ae72-8b091238cc2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:01:20.712663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.712627 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-kube-api-access-4vpf9" pod "7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" (UID: "7bbcaa62-d63a-4bf4-ae72-8b091238cc2f"). InnerVolumeSpecName "kube-api-access-4vpf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:20.713161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.713136 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-envoy" pod "0b6f67a0-2dae-400a-abd7-7c65fbdac225" (UID: "0b6f67a0-2dae-400a-abd7-7c65fbdac225"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:20.713658 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.713627 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-token" pod "7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" (UID: "7bbcaa62-d63a-4bf4-ae72-8b091238cc2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:20.713760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.713741 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-podinfo" pod "0b6f67a0-2dae-400a-abd7-7c65fbdac225" (UID: "0b6f67a0-2dae-400a-abd7-7c65fbdac225"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:01:20.809260 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.809221 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.809260 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.809254 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.809260 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.809265 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.809578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.809274 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.809578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.809283 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.809578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.809292 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.809578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.809299 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.809578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.809307 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.809578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.809316 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.809578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.809324 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vpf9\" (UniqueName: \"kubernetes.io/projected/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-kube-api-access-4vpf9\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.809578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.809332 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwlkn\" (UniqueName: \"kubernetes.io/projected/0b6f67a0-2dae-400a-abd7-7c65fbdac225-kube-api-access-cwlkn\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.809578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.809340 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.809578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.809349 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:20.809578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:20.809356 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b6f67a0-2dae-400a-abd7-7c65fbdac225-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:21.058142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.058023 5998 generic.go:356] "Generic (PLEG): container finished" podID="a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" containerID="aef65e1106b7ffb44a2e604aa32084565c216447430dc8fd93258f188c4b23a9" exitCode=0 Oct 30 02:01:21.058142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.058072 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" Oct 30 02:01:21.058142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.058118 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" event={"ID":"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5","Type":"ContainerDied","Data":"aef65e1106b7ffb44a2e604aa32084565c216447430dc8fd93258f188c4b23a9"} Oct 30 02:01:21.058453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.058189 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn" event={"ID":"a9c105b9-9c67-48ab-821c-b16c9f9ef5e5","Type":"ContainerDied","Data":"b449ac37823b6e933bf6bde8809b0caae18acd134843e4c6b8e0d8734f1b27da"} Oct 30 02:01:21.058453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.058225 5998 scope.go:122] "RemoveContainer" containerID="aef65e1106b7ffb44a2e604aa32084565c216447430dc8fd93258f188c4b23a9" Oct 30 02:01:21.059585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.059564 5998 generic.go:356] "Generic (PLEG): container finished" podID="7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" containerID="1f20376a1a3e8e4701c80f9584c3e1b1080554df0fa72ddaa99af2a76d5bf4dc" exitCode=0 Oct 30 02:01:21.059696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.059599 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" event={"ID":"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f","Type":"ContainerDied","Data":"1f20376a1a3e8e4701c80f9584c3e1b1080554df0fa72ddaa99af2a76d5bf4dc"} Oct 30 02:01:21.059696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.059619 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" Oct 30 02:01:21.059696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.059652 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t" event={"ID":"7bbcaa62-d63a-4bf4-ae72-8b091238cc2f","Type":"ContainerDied","Data":"8498b798e9d19b08521afccdbe5b6a380b59304a63836ec7cf015089430a6bd6"} Oct 30 02:01:21.061169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.061149 5998 generic.go:356] "Generic (PLEG): container finished" podID="3605446f-2664-4b96-9a9f-e13aab64faee" containerID="883b0ca245b89825ab2c2b7bccd313bc9c11f9633acbaafebbaccfb62607a603" exitCode=0 Oct 30 02:01:21.061259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.061208 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" Oct 30 02:01:21.061259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.061234 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" event={"ID":"3605446f-2664-4b96-9a9f-e13aab64faee","Type":"ContainerDied","Data":"883b0ca245b89825ab2c2b7bccd313bc9c11f9633acbaafebbaccfb62607a603"} Oct 30 02:01:21.061337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.061299 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w" event={"ID":"3605446f-2664-4b96-9a9f-e13aab64faee","Type":"ContainerDied","Data":"4f029ce0ddfbac192dc08b989105f68f1e167389c932199620d1764b2bd9004e"} Oct 30 02:01:21.062795 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.062772 5998 generic.go:356] "Generic (PLEG): container finished" podID="0b6f67a0-2dae-400a-abd7-7c65fbdac225" containerID="3d85368bb06bb388c8f248f640ccf045c6d5eed581fad5fbf75b276628ea0ba0" exitCode=0 Oct 30 02:01:21.062880 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.062811 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" Oct 30 02:01:21.062945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.062805 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" event={"ID":"0b6f67a0-2dae-400a-abd7-7c65fbdac225","Type":"ContainerDied","Data":"3d85368bb06bb388c8f248f640ccf045c6d5eed581fad5fbf75b276628ea0ba0"} Oct 30 02:01:21.062996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.062949 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm" event={"ID":"0b6f67a0-2dae-400a-abd7-7c65fbdac225","Type":"ContainerDied","Data":"08470e6ee30bb2d68b34d7d0481ec81555890caeb3f706d2740b4ce6f367f192"} Oct 30 02:01:21.072297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.072271 5998 scope.go:122] "RemoveContainer" containerID="aef65e1106b7ffb44a2e604aa32084565c216447430dc8fd93258f188c4b23a9" Oct 30 02:01:21.072664 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:01:21.072615 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef65e1106b7ffb44a2e604aa32084565c216447430dc8fd93258f188c4b23a9\": container with ID starting with aef65e1106b7ffb44a2e604aa32084565c216447430dc8fd93258f188c4b23a9 not found: ID does not exist" containerID="aef65e1106b7ffb44a2e604aa32084565c216447430dc8fd93258f188c4b23a9" Oct 30 02:01:21.072664 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.072649 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef65e1106b7ffb44a2e604aa32084565c216447430dc8fd93258f188c4b23a9"} err="failed to get container status \"aef65e1106b7ffb44a2e604aa32084565c216447430dc8fd93258f188c4b23a9\": rpc error: code = NotFound desc = could not find container \"aef65e1106b7ffb44a2e604aa32084565c216447430dc8fd93258f188c4b23a9\": container with ID starting with aef65e1106b7ffb44a2e604aa32084565c216447430dc8fd93258f188c4b23a9 not found: ID does not exist" Oct 30 02:01:21.072799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.072675 5998 scope.go:122] "RemoveContainer" containerID="1f20376a1a3e8e4701c80f9584c3e1b1080554df0fa72ddaa99af2a76d5bf4dc" Oct 30 02:01:21.086054 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.086033 5998 scope.go:122] "RemoveContainer" containerID="1f20376a1a3e8e4701c80f9584c3e1b1080554df0fa72ddaa99af2a76d5bf4dc" Oct 30 02:01:21.087561 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:01:21.086393 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f20376a1a3e8e4701c80f9584c3e1b1080554df0fa72ddaa99af2a76d5bf4dc\": container with ID starting with 1f20376a1a3e8e4701c80f9584c3e1b1080554df0fa72ddaa99af2a76d5bf4dc not found: ID does not exist" containerID="1f20376a1a3e8e4701c80f9584c3e1b1080554df0fa72ddaa99af2a76d5bf4dc" Oct 30 02:01:21.087561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.086426 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f20376a1a3e8e4701c80f9584c3e1b1080554df0fa72ddaa99af2a76d5bf4dc"} err="failed to get container status \"1f20376a1a3e8e4701c80f9584c3e1b1080554df0fa72ddaa99af2a76d5bf4dc\": rpc error: code = NotFound desc = could not find container \"1f20376a1a3e8e4701c80f9584c3e1b1080554df0fa72ddaa99af2a76d5bf4dc\": container with ID starting with 1f20376a1a3e8e4701c80f9584c3e1b1080554df0fa72ddaa99af2a76d5bf4dc not found: ID does not exist" Oct 30 02:01:21.087561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.086444 5998 scope.go:122] "RemoveContainer" containerID="883b0ca245b89825ab2c2b7bccd313bc9c11f9633acbaafebbaccfb62607a603" Oct 30 02:01:21.096632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.096616 5998 scope.go:122] "RemoveContainer" containerID="883b0ca245b89825ab2c2b7bccd313bc9c11f9633acbaafebbaccfb62607a603" Oct 30 02:01:21.096942 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:01:21.096920 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"883b0ca245b89825ab2c2b7bccd313bc9c11f9633acbaafebbaccfb62607a603\": container with ID starting with 883b0ca245b89825ab2c2b7bccd313bc9c11f9633acbaafebbaccfb62607a603 not found: ID does not exist" containerID="883b0ca245b89825ab2c2b7bccd313bc9c11f9633acbaafebbaccfb62607a603" Oct 30 02:01:21.097025 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.096953 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"883b0ca245b89825ab2c2b7bccd313bc9c11f9633acbaafebbaccfb62607a603"} err="failed to get container status \"883b0ca245b89825ab2c2b7bccd313bc9c11f9633acbaafebbaccfb62607a603\": rpc error: code = NotFound desc = could not find container \"883b0ca245b89825ab2c2b7bccd313bc9c11f9633acbaafebbaccfb62607a603\": container with ID starting with 883b0ca245b89825ab2c2b7bccd313bc9c11f9633acbaafebbaccfb62607a603 not found: ID does not exist" Oct 30 02:01:21.097025 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.096975 5998 scope.go:122] "RemoveContainer" containerID="3d85368bb06bb388c8f248f640ccf045c6d5eed581fad5fbf75b276628ea0ba0" Oct 30 02:01:21.109258 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.109238 5998 scope.go:122] "RemoveContainer" containerID="3d85368bb06bb388c8f248f640ccf045c6d5eed581fad5fbf75b276628ea0ba0" Oct 30 02:01:21.109558 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:01:21.109541 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d85368bb06bb388c8f248f640ccf045c6d5eed581fad5fbf75b276628ea0ba0\": container with ID starting with 3d85368bb06bb388c8f248f640ccf045c6d5eed581fad5fbf75b276628ea0ba0 not found: ID does not exist" containerID="3d85368bb06bb388c8f248f640ccf045c6d5eed581fad5fbf75b276628ea0ba0" Oct 30 02:01:21.109620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.109566 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d85368bb06bb388c8f248f640ccf045c6d5eed581fad5fbf75b276628ea0ba0"} err="failed to get container status \"3d85368bb06bb388c8f248f640ccf045c6d5eed581fad5fbf75b276628ea0ba0\": rpc error: code = NotFound desc = could not find container \"3d85368bb06bb388c8f248f640ccf045c6d5eed581fad5fbf75b276628ea0ba0\": container with ID starting with 3d85368bb06bb388c8f248f640ccf045c6d5eed581fad5fbf75b276628ea0ba0 not found: ID does not exist" Oct 30 02:01:21.175520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.175481 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t"] Oct 30 02:01:21.176973 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.176943 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:21.186884 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.186849 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-vj42t"] Oct 30 02:01:21.216744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.213446 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn"] Oct 30 02:01:21.218623 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.218592 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-qz4gn"] Oct 30 02:01:21.252181 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.252148 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w"] Oct 30 02:01:21.256901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.256872 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-5cw4w"] Oct 30 02:01:21.285594 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.285557 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm"] Oct 30 02:01:21.292914 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:21.292880 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hfrm"] Oct 30 02:01:22.208343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:22.208307 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:22.540306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:22.540212 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6f67a0-2dae-400a-abd7-7c65fbdac225" path="/var/lib/kubelet/pods/0b6f67a0-2dae-400a-abd7-7c65fbdac225/volumes" Oct 30 02:01:22.540828 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:22.540805 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3605446f-2664-4b96-9a9f-e13aab64faee" path="/var/lib/kubelet/pods/3605446f-2664-4b96-9a9f-e13aab64faee/volumes" Oct 30 02:01:22.541338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:22.541319 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" path="/var/lib/kubelet/pods/7bbcaa62-d63a-4bf4-ae72-8b091238cc2f/volumes" Oct 30 02:01:22.542131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:22.542080 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" path="/var/lib/kubelet/pods/a9c105b9-9c67-48ab-821c-b16c9f9ef5e5/volumes" Oct 30 02:01:23.727521 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:23.727481 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:23.885971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:23.885932 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:24.056118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:24.056006 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:24.377227 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:24.377144 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:24.488346 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:24.488311 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:24.735121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:24.734471 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:24.790152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:24.790114 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76"] Oct 30 02:01:24.790431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:24.790389 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" podUID="323ecfe9-7182-4e29-9774-dd3dd728fc09" containerName="istio-proxy" containerID="cri-o://bcf0f46b790721606aa05544dca4b763434d28b0309cc6d082ba7e37a62c34c3" gracePeriod=30 Oct 30 02:01:24.856970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:24.856930 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:25.226619 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:01:25.226580 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:01:25.587144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:25.587044 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:26.112003 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:26.111967 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:27.503845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:27.503810 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:27.616937 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:27.616902 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:28.471590 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:28.471551 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:30.106665 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.106621 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:30.110104 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.110065 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 02:01:30.160330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.158357 5998 generic.go:356] "Generic (PLEG): container finished" podID="323ecfe9-7182-4e29-9774-dd3dd728fc09" containerID="bcf0f46b790721606aa05544dca4b763434d28b0309cc6d082ba7e37a62c34c3" exitCode=0 Oct 30 02:01:30.160330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.158459 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" event={"ID":"323ecfe9-7182-4e29-9774-dd3dd728fc09","Type":"ContainerDied","Data":"bcf0f46b790721606aa05544dca4b763434d28b0309cc6d082ba7e37a62c34c3"} Oct 30 02:01:30.160330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.158517 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" event={"ID":"323ecfe9-7182-4e29-9774-dd3dd728fc09","Type":"ContainerDied","Data":"fe3e1d9aa330adef2317a80ce920358909c7b4e10aa880d2a3587b93ac307be5"} Oct 30 02:01:30.160330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.158548 5998 scope.go:122] "RemoveContainer" containerID="bcf0f46b790721606aa05544dca4b763434d28b0309cc6d082ba7e37a62c34c3" Oct 30 02:01:30.160330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.158722 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76" Oct 30 02:01:30.163915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.163851 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-data\") pod \"323ecfe9-7182-4e29-9774-dd3dd728fc09\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " Oct 30 02:01:30.172177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.164078 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-podinfo\") pod \"323ecfe9-7182-4e29-9774-dd3dd728fc09\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " Oct 30 02:01:30.174666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.172654 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/323ecfe9-7182-4e29-9774-dd3dd728fc09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323ecfe9-7182-4e29-9774-dd3dd728fc09-istiod-ca-cert\") pod \"323ecfe9-7182-4e29-9774-dd3dd728fc09\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " Oct 30 02:01:30.174666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.172726 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-workload-certs\") pod \"323ecfe9-7182-4e29-9774-dd3dd728fc09\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " Oct 30 02:01:30.174666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.172759 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-envoy\") pod \"323ecfe9-7182-4e29-9774-dd3dd728fc09\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " Oct 30 02:01:30.174666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.172825 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-workload-socket\") pod \"323ecfe9-7182-4e29-9774-dd3dd728fc09\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " Oct 30 02:01:30.174666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.172880 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-credential-socket\") pod \"323ecfe9-7182-4e29-9774-dd3dd728fc09\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " Oct 30 02:01:30.174666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.172929 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323ecfe9-7182-4e29-9774-dd3dd728fc09-kube-api-access-rrblp\" (UniqueName: \"kubernetes.io/projected/323ecfe9-7182-4e29-9774-dd3dd728fc09-kube-api-access-rrblp\") pod \"323ecfe9-7182-4e29-9774-dd3dd728fc09\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " Oct 30 02:01:30.174666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.172959 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-token\" (UniqueName: \"kubernetes.io/projected/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-token\") pod \"323ecfe9-7182-4e29-9774-dd3dd728fc09\" (UID: \"323ecfe9-7182-4e29-9774-dd3dd728fc09\") " Oct 30 02:01:30.188174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.175573 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-workload-socket" pod "323ecfe9-7182-4e29-9774-dd3dd728fc09" (UID: "323ecfe9-7182-4e29-9774-dd3dd728fc09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:30.188174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.181871 5998 scope.go:122] "RemoveContainer" containerID="aa45e4b085d2175b53a045d6483a965484aea40259e90f4dabb71fea592559e1" Oct 30 02:01:30.188174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.183399 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-credential-socket" pod "323ecfe9-7182-4e29-9774-dd3dd728fc09" (UID: "323ecfe9-7182-4e29-9774-dd3dd728fc09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:30.230895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.196879 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323ecfe9-7182-4e29-9774-dd3dd728fc09-kube-api-access-rrblp" pod "323ecfe9-7182-4e29-9774-dd3dd728fc09" (UID: "323ecfe9-7182-4e29-9774-dd3dd728fc09"). InnerVolumeSpecName "kube-api-access-rrblp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:30.230895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.199311 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-podinfo" pod "323ecfe9-7182-4e29-9774-dd3dd728fc09" (UID: "323ecfe9-7182-4e29-9774-dd3dd728fc09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:01:30.230895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.214462 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-data" pod "323ecfe9-7182-4e29-9774-dd3dd728fc09" (UID: "323ecfe9-7182-4e29-9774-dd3dd728fc09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:30.230895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.215278 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323ecfe9-7182-4e29-9774-dd3dd728fc09-istiod-ca-cert" pod "323ecfe9-7182-4e29-9774-dd3dd728fc09" (UID: "323ecfe9-7182-4e29-9774-dd3dd728fc09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:30.230895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.218347 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-workload-certs" pod "323ecfe9-7182-4e29-9774-dd3dd728fc09" (UID: "323ecfe9-7182-4e29-9774-dd3dd728fc09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:30.262717 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.262674 5998 scope.go:122] "RemoveContainer" containerID="bcf0f46b790721606aa05544dca4b763434d28b0309cc6d082ba7e37a62c34c3" Oct 30 02:01:30.262946 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.262921 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-token" pod "323ecfe9-7182-4e29-9774-dd3dd728fc09" (UID: "323ecfe9-7182-4e29-9774-dd3dd728fc09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:30.263420 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:01:30.263382 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf0f46b790721606aa05544dca4b763434d28b0309cc6d082ba7e37a62c34c3\": container with ID starting with bcf0f46b790721606aa05544dca4b763434d28b0309cc6d082ba7e37a62c34c3 not found: ID does not exist" containerID="bcf0f46b790721606aa05544dca4b763434d28b0309cc6d082ba7e37a62c34c3" Oct 30 02:01:30.263535 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.263422 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf0f46b790721606aa05544dca4b763434d28b0309cc6d082ba7e37a62c34c3"} err="failed to get container status \"bcf0f46b790721606aa05544dca4b763434d28b0309cc6d082ba7e37a62c34c3\": rpc error: code = NotFound desc = could not find container \"bcf0f46b790721606aa05544dca4b763434d28b0309cc6d082ba7e37a62c34c3\": container with ID starting with bcf0f46b790721606aa05544dca4b763434d28b0309cc6d082ba7e37a62c34c3 not found: ID does not exist" Oct 30 02:01:30.263535 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.263449 5998 scope.go:122] "RemoveContainer" containerID="aa45e4b085d2175b53a045d6483a965484aea40259e90f4dabb71fea592559e1" Oct 30 02:01:30.263749 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:01:30.263726 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa45e4b085d2175b53a045d6483a965484aea40259e90f4dabb71fea592559e1\": container with ID starting with aa45e4b085d2175b53a045d6483a965484aea40259e90f4dabb71fea592559e1 not found: ID does not exist" containerID="aa45e4b085d2175b53a045d6483a965484aea40259e90f4dabb71fea592559e1" Oct 30 02:01:30.263819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.263751 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa45e4b085d2175b53a045d6483a965484aea40259e90f4dabb71fea592559e1"} err="failed to get container status \"aa45e4b085d2175b53a045d6483a965484aea40259e90f4dabb71fea592559e1\": rpc error: code = NotFound desc = could not find container \"aa45e4b085d2175b53a045d6483a965484aea40259e90f4dabb71fea592559e1\": container with ID starting with aa45e4b085d2175b53a045d6483a965484aea40259e90f4dabb71fea592559e1 not found: ID does not exist" Oct 30 02:01:30.264721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.264678 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-envoy" pod "323ecfe9-7182-4e29-9774-dd3dd728fc09" (UID: "323ecfe9-7182-4e29-9774-dd3dd728fc09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:01:30.274708 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.274680 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/323ecfe9-7182-4e29-9774-dd3dd728fc09-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:30.274708 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.274709 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:30.274855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.274718 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:30.274855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.274730 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:30.274855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.274738 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:30.274855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.274747 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrblp\" (UniqueName: \"kubernetes.io/projected/323ecfe9-7182-4e29-9774-dd3dd728fc09-kube-api-access-rrblp\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:30.274855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.274756 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:30.274855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.274765 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:30.274855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.274773 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/323ecfe9-7182-4e29-9774-dd3dd728fc09-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:30.515073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.511337 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76"] Oct 30 02:01:30.519831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.519798 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b273b6-openshift-default-55d489cc49-ffl76"] Oct 30 02:01:30.542278 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:30.542244 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="323ecfe9-7182-4e29-9774-dd3dd728fc09" path="/var/lib/kubelet/pods/323ecfe9-7182-4e29-9774-dd3dd728fc09/volumes" Oct 30 02:01:31.323615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:31.323582 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:01:31.559220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:31.559185 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:31.693000 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:31.692953 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w"] Oct 30 02:01:31.693263 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:31.693230 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" podUID="4f80e088-78c9-4b95-98cc-4d1dd67d9a34" containerName="payload-pre-processing" containerID="cri-o://d7e83251aa9f12aefcf1bceefb3fb3461812edcb31fdffb834c6f1bcd29d7188" gracePeriod=30 Oct 30 02:01:31.750668 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:31.749930 5998 scope.go:122] "RemoveContainer" containerID="9d67d9deb181e72c75fe8aeca76f9b04c9ef180b888541418e05ae395271ef79" Oct 30 02:01:31.750668 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:31.750439 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:31.792914 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:31.792863 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" podUID="4f80e088-78c9-4b95-98cc-4d1dd67d9a34" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.105:9004: connect: connection refused" Oct 30 02:01:31.816308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:31.815613 5998 scope.go:122] "RemoveContainer" containerID="6760573b71328f8026a62594424f04a19961483c33ee2d87d11ab255021046e4" Oct 30 02:01:31.864009 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:31.863975 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:32.047277 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.047252 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" Oct 30 02:01:32.131254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.115668 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f80e088-78c9-4b95-98cc-4d1dd67d9a34-kube-api-access-97clm\" (UniqueName: \"kubernetes.io/projected/4f80e088-78c9-4b95-98cc-4d1dd67d9a34-kube-api-access-97clm\") pod \"4f80e088-78c9-4b95-98cc-4d1dd67d9a34\" (UID: \"4f80e088-78c9-4b95-98cc-4d1dd67d9a34\") " Oct 30 02:01:32.131254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.115764 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f80e088-78c9-4b95-98cc-4d1dd67d9a34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f80e088-78c9-4b95-98cc-4d1dd67d9a34-plugins-config-volume\") pod \"4f80e088-78c9-4b95-98cc-4d1dd67d9a34\" (UID: \"4f80e088-78c9-4b95-98cc-4d1dd67d9a34\") " Oct 30 02:01:32.150471 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.138968 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f80e088-78c9-4b95-98cc-4d1dd67d9a34-kube-api-access-97clm" pod "4f80e088-78c9-4b95-98cc-4d1dd67d9a34" (UID: "4f80e088-78c9-4b95-98cc-4d1dd67d9a34"). InnerVolumeSpecName "kube-api-access-97clm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:32.150471 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.139021 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f80e088-78c9-4b95-98cc-4d1dd67d9a34-plugins-config-volume" pod "4f80e088-78c9-4b95-98cc-4d1dd67d9a34" (UID: "4f80e088-78c9-4b95-98cc-4d1dd67d9a34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:32.227804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.226843 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97clm\" (UniqueName: \"kubernetes.io/projected/4f80e088-78c9-4b95-98cc-4d1dd67d9a34-kube-api-access-97clm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:32.227804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.226881 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f80e088-78c9-4b95-98cc-4d1dd67d9a34-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:01:32.255349 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.255315 5998 generic.go:356] "Generic (PLEG): container finished" podID="4f80e088-78c9-4b95-98cc-4d1dd67d9a34" containerID="d7e83251aa9f12aefcf1bceefb3fb3461812edcb31fdffb834c6f1bcd29d7188" exitCode=0 Oct 30 02:01:32.255547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.255405 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" event={"ID":"4f80e088-78c9-4b95-98cc-4d1dd67d9a34","Type":"ContainerDied","Data":"d7e83251aa9f12aefcf1bceefb3fb3461812edcb31fdffb834c6f1bcd29d7188"} Oct 30 02:01:32.255547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.255442 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" event={"ID":"4f80e088-78c9-4b95-98cc-4d1dd67d9a34","Type":"ContainerDied","Data":"02a8282d351060c0d014ad71d12824317a0d6e1c5cb9529698e8fe7893ee9ced"} Oct 30 02:01:32.255547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.255468 5998 scope.go:122] "RemoveContainer" containerID="d7e83251aa9f12aefcf1bceefb3fb3461812edcb31fdffb834c6f1bcd29d7188" Oct 30 02:01:32.255715 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.255576 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w" Oct 30 02:01:32.282952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.282835 5998 scope.go:122] "RemoveContainer" containerID="d7e83251aa9f12aefcf1bceefb3fb3461812edcb31fdffb834c6f1bcd29d7188" Oct 30 02:01:32.283322 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:01:32.283291 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e83251aa9f12aefcf1bceefb3fb3461812edcb31fdffb834c6f1bcd29d7188\": container with ID starting with d7e83251aa9f12aefcf1bceefb3fb3461812edcb31fdffb834c6f1bcd29d7188 not found: ID does not exist" containerID="d7e83251aa9f12aefcf1bceefb3fb3461812edcb31fdffb834c6f1bcd29d7188" Oct 30 02:01:32.283448 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.283330 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e83251aa9f12aefcf1bceefb3fb3461812edcb31fdffb834c6f1bcd29d7188"} err="failed to get container status \"d7e83251aa9f12aefcf1bceefb3fb3461812edcb31fdffb834c6f1bcd29d7188\": rpc error: code = NotFound desc = could not find container \"d7e83251aa9f12aefcf1bceefb3fb3461812edcb31fdffb834c6f1bcd29d7188\": container with ID starting with d7e83251aa9f12aefcf1bceefb3fb3461812edcb31fdffb834c6f1bcd29d7188 not found: ID does not exist" Oct 30 02:01:32.310960 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.310924 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w"] Oct 30 02:01:32.315441 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.315408 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7542fa-c4f7789b7-kxf7w"] Oct 30 02:01:32.410219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.410186 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:32.539453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.539373 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f80e088-78c9-4b95-98cc-4d1dd67d9a34" path="/var/lib/kubelet/pods/4f80e088-78c9-4b95-98cc-4d1dd67d9a34/volumes" Oct 30 02:01:32.967313 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:32.967268 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:33.081471 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:33.078805 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:33.746543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:33.746510 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:34.059005 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:34.058909 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:34.165555 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:34.165518 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:34.793497 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:34.793460 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:35.022864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:35.022827 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:35.237471 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:35.237438 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:35.583118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:35.583004 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:36.037818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:36.037785 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:37.309328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:37.309291 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:38.932556 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:38.932523 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:40.037820 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:40.037783 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc"] Oct 30 02:01:40.049308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:40.049274 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e82ba923-zwlpc"] Oct 30 02:01:40.545666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:40.545628 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c79834ac-25b6-4c70-b68f-2ee4228f6001" path="/var/lib/kubelet/pods/c79834ac-25b6-4c70-b68f-2ee4228f6001/volumes" Oct 30 02:01:40.545890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:40.545863 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:40.658876 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:40.658841 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:40.788472 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:40.788434 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:41.671893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:41.671860 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:42.192553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:42.192496 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:43.483025 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:43.482996 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:43.875020 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:43.874934 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:45.620350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:45.620313 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:46.812334 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:46.812297 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:46.954943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:46.954906 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:47.929820 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:47.929778 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:49.112579 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:49.112535 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:49.410171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:49.396851 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:50.270703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:50.270670 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:51.892645 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:51.892613 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:53.748929 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:53.748887 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:54.036165 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:54.035355 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd"] Oct 30 02:01:54.067011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:54.066973 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b769c4-5v2rd"] Oct 30 02:01:54.539885 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:54.539843 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f" path="/var/lib/kubelet/pods/e6c5a893-d76d-417e-b3ec-76d6e5fc5d4f/volumes" Oct 30 02:01:54.541668 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:54.541640 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:54.743116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:54.733514 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:01:55.300151 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:01:55.300120 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:01:56.325352 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:56.325310 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:56.487050 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:56.487014 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:56.689113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:56.688168 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:57.217633 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:57.217583 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:58.412380 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:58.412347 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:58.519952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:58.519895 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:58.635642 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:58.635605 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:59.167727 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:59.167665 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:59.560304 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:59.560200 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:01:59.672100 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:01:59.672044 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:00.480078 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:00.480047 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:00.631718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:00.631681 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:00.747659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:00.747519 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:00.979177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:00.979139 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:01.102457 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:01.102354 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:01.535552 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:01.535504 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:01.783701 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:01.783652 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:01.797606 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:01.797509 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv"] Oct 30 02:02:01.797831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:01.797800 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" podUID="3daf0043-f037-417f-bf48-7a8f53565395" containerName="istio-proxy" containerID="cri-o://880499a7aa319b9434fc6e54076a0df60466d2e0f7f3221653263a8715ef8116" gracePeriod=30 Oct 30 02:02:01.894133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:01.894044 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:03.324708 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:03.324669 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:03.596699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:03.596584 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:03.888173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:03.876987 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:04.033722 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:04.033660 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:04.684413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:04.684373 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:04.816076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:04.816000 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:05.424720 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:05.424664 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:05.752835 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:05.752691 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:06.730555 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:06.730504 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:07.116574 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.116543 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:02:07.280661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.280583 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-workload-certs\") pod \"3daf0043-f037-417f-bf48-7a8f53565395\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " Oct 30 02:02:07.280904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.280877 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3daf0043-f037-417f-bf48-7a8f53565395-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daf0043-f037-417f-bf48-7a8f53565395-istio-podinfo\") pod \"3daf0043-f037-417f-bf48-7a8f53565395\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " Oct 30 02:02:07.280904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.280893 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-workload-certs" pod "3daf0043-f037-417f-bf48-7a8f53565395" (UID: "3daf0043-f037-417f-bf48-7a8f53565395"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.281124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.280941 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3daf0043-f037-417f-bf48-7a8f53565395-kube-api-access-4h5nk\" (UniqueName: \"kubernetes.io/projected/3daf0043-f037-417f-bf48-7a8f53565395-kube-api-access-4h5nk\") pod \"3daf0043-f037-417f-bf48-7a8f53565395\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " Oct 30 02:02:07.281124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.281023 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3daf0043-f037-417f-bf48-7a8f53565395-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daf0043-f037-417f-bf48-7a8f53565395-istiod-ca-cert\") pod \"3daf0043-f037-417f-bf48-7a8f53565395\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " Oct 30 02:02:07.281124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.281057 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-workload-socket\") pod \"3daf0043-f037-417f-bf48-7a8f53565395\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " Oct 30 02:02:07.281124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.281079 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-istio-data\") pod \"3daf0043-f037-417f-bf48-7a8f53565395\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " Oct 30 02:02:07.281366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.281214 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-istio-envoy\") pod \"3daf0043-f037-417f-bf48-7a8f53565395\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " Oct 30 02:02:07.281366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.281259 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-credential-socket\") pod \"3daf0043-f037-417f-bf48-7a8f53565395\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " Oct 30 02:02:07.281366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.281291 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3daf0043-f037-417f-bf48-7a8f53565395-istio-token\" (UniqueName: \"kubernetes.io/projected/3daf0043-f037-417f-bf48-7a8f53565395-istio-token\") pod \"3daf0043-f037-417f-bf48-7a8f53565395\" (UID: \"3daf0043-f037-417f-bf48-7a8f53565395\") " Oct 30 02:02:07.281849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.281786 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-istio-data" pod "3daf0043-f037-417f-bf48-7a8f53565395" (UID: "3daf0043-f037-417f-bf48-7a8f53565395"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.281985 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.281868 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.282171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.282144 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3daf0043-f037-417f-bf48-7a8f53565395-istiod-ca-cert" pod "3daf0043-f037-417f-bf48-7a8f53565395" (UID: "3daf0043-f037-417f-bf48-7a8f53565395"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:07.282266 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.282233 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-credential-socket" pod "3daf0043-f037-417f-bf48-7a8f53565395" (UID: "3daf0043-f037-417f-bf48-7a8f53565395"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.282513 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.282480 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-workload-socket" pod "3daf0043-f037-417f-bf48-7a8f53565395" (UID: "3daf0043-f037-417f-bf48-7a8f53565395"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.283935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.283913 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3daf0043-f037-417f-bf48-7a8f53565395-kube-api-access-4h5nk" pod "3daf0043-f037-417f-bf48-7a8f53565395" (UID: "3daf0043-f037-417f-bf48-7a8f53565395"). InnerVolumeSpecName "kube-api-access-4h5nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:07.284171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.284148 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3daf0043-f037-417f-bf48-7a8f53565395-istio-token" pod "3daf0043-f037-417f-bf48-7a8f53565395" (UID: "3daf0043-f037-417f-bf48-7a8f53565395"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:07.284794 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.284769 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3daf0043-f037-417f-bf48-7a8f53565395-istio-podinfo" pod "3daf0043-f037-417f-bf48-7a8f53565395" (UID: "3daf0043-f037-417f-bf48-7a8f53565395"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:02:07.284885 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.284843 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-istio-envoy" pod "3daf0043-f037-417f-bf48-7a8f53565395" (UID: "3daf0043-f037-417f-bf48-7a8f53565395"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:07.382877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.382843 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daf0043-f037-417f-bf48-7a8f53565395-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.382877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.382873 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h5nk\" (UniqueName: \"kubernetes.io/projected/3daf0043-f037-417f-bf48-7a8f53565395-kube-api-access-4h5nk\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.382877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.382883 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daf0043-f037-417f-bf48-7a8f53565395-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.383154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.382898 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.383154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.382909 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.383154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.382917 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.383154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.382924 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0043-f037-417f-bf48-7a8f53565395-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.383154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.382931 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3daf0043-f037-417f-bf48-7a8f53565395-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:07.606006 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.605920 5998 generic.go:356] "Generic (PLEG): container finished" podID="3daf0043-f037-417f-bf48-7a8f53565395" containerID="880499a7aa319b9434fc6e54076a0df60466d2e0f7f3221653263a8715ef8116" exitCode=0 Oct 30 02:02:07.606349 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.606314 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" event={"ID":"3daf0043-f037-417f-bf48-7a8f53565395","Type":"ContainerDied","Data":"880499a7aa319b9434fc6e54076a0df60466d2e0f7f3221653263a8715ef8116"} Oct 30 02:02:07.606478 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.606402 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" event={"ID":"3daf0043-f037-417f-bf48-7a8f53565395","Type":"ContainerDied","Data":"c216991d46a7b1949107e031c52b43465132a3d965e51c21f6db00e3421ce16a"} Oct 30 02:02:07.606478 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.606440 5998 scope.go:122] "RemoveContainer" containerID="880499a7aa319b9434fc6e54076a0df60466d2e0f7f3221653263a8715ef8116" Oct 30 02:02:07.606683 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.606291 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv" Oct 30 02:02:07.624281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.624256 5998 scope.go:122] "RemoveContainer" containerID="880499a7aa319b9434fc6e54076a0df60466d2e0f7f3221653263a8715ef8116" Oct 30 02:02:07.624702 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:02:07.624665 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880499a7aa319b9434fc6e54076a0df60466d2e0f7f3221653263a8715ef8116\": container with ID starting with 880499a7aa319b9434fc6e54076a0df60466d2e0f7f3221653263a8715ef8116 not found: ID does not exist" containerID="880499a7aa319b9434fc6e54076a0df60466d2e0f7f3221653263a8715ef8116" Oct 30 02:02:07.624868 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.624695 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880499a7aa319b9434fc6e54076a0df60466d2e0f7f3221653263a8715ef8116"} err="failed to get container status \"880499a7aa319b9434fc6e54076a0df60466d2e0f7f3221653263a8715ef8116\": rpc error: code = NotFound desc = could not find container \"880499a7aa319b9434fc6e54076a0df60466d2e0f7f3221653263a8715ef8116\": container with ID starting with 880499a7aa319b9434fc6e54076a0df60466d2e0f7f3221653263a8715ef8116 not found: ID does not exist" Oct 30 02:02:07.663411 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.663368 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:07.670562 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.670529 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv"] Oct 30 02:02:07.676563 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.675145 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ba1d9-openshift-default-587457755f-rlcdv"] Oct 30 02:02:07.787316 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.787282 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:07.920820 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:07.920778 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:08.396302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:08.396219 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:08.593113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:08.590293 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3daf0043-f037-417f-bf48-7a8f53565395" path="/var/lib/kubelet/pods/3daf0043-f037-417f-bf48-7a8f53565395/volumes" Oct 30 02:02:08.855706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:08.855669 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:09.772259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:09.772225 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:09.905703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:09.905668 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:12.396966 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:12.396934 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:12.824510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:12.824474 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:12.896890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:12.896856 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5"] Oct 30 02:02:12.897115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:12.897076 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" podUID="6cd04c1d-6bb4-4cd8-99b5-523538908019" containerName="istio-proxy" containerID="cri-o://a910a7c81dc5da24505757cd5dabbad04e5a5cdd19a3122057bdef721cae0c85" gracePeriod=30 Oct 30 02:02:12.911008 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:12.910930 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq"] Oct 30 02:02:12.911250 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:12.911226 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" podUID="63830843-4d31-45a7-83c9-9bb2248abe3b" containerName="istio-proxy" containerID="cri-o://0f2bd476b2115992a343a857c0baab5dbec6e4121a745c82ad3e0e24a24db26e" gracePeriod=30 Oct 30 02:02:12.914438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:12.914413 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk"] Oct 30 02:02:12.914686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:12.914657 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" podUID="94053f0f-2fd8-4ab0-914a-256ddadb5d85" containerName="istio-proxy" containerID="cri-o://3027f63794dc2178391b57d4f2c91a0592db3cf295ea9b87658997a8e772778b" gracePeriod=30 Oct 30 02:02:13.011180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:13.011142 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:14.353961 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:14.353927 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:14.564011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:14.563954 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:14.672418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:14.672371 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:15.100279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:15.100196 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:16.608289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:16.608255 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:16.784149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:16.784109 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:16.891896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:16.891814 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:17.449848 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:17.449812 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:18.266833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.266802 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 02:02:18.293306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.293284 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 02:02:18.297732 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.297709 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 02:02:18.326271 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.326161 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-workload-certs\") pod \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " Oct 30 02:02:18.326271 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.326257 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cd04c1d-6bb4-4cd8-99b5-523538908019-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd04c1d-6bb4-4cd8-99b5-523538908019-istiod-ca-cert\") pod \"6cd04c1d-6bb4-4cd8-99b5-523538908019\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " Oct 30 02:02:18.327239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.326288 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-data\") pod \"63830843-4d31-45a7-83c9-9bb2248abe3b\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " Oct 30 02:02:18.327239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.326324 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-workload-socket\") pod \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " Oct 30 02:02:18.327239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.326830 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-workload-certs" pod "94053f0f-2fd8-4ab0-914a-256ddadb5d85" (UID: "94053f0f-2fd8-4ab0-914a-256ddadb5d85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:18.327239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.326940 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-token\" (UniqueName: \"kubernetes.io/projected/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-token\") pod \"6cd04c1d-6bb4-4cd8-99b5-523538908019\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " Oct 30 02:02:18.327239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.326987 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-envoy\") pod \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " Oct 30 02:02:18.327239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327017 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-credential-socket\") pod \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " Oct 30 02:02:18.327239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327065 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-envoy\") pod \"63830843-4d31-45a7-83c9-9bb2248abe3b\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " Oct 30 02:02:18.327239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327109 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-data\") pod \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " Oct 30 02:02:18.327239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327150 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63830843-4d31-45a7-83c9-9bb2248abe3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63830843-4d31-45a7-83c9-9bb2248abe3b-istiod-ca-cert\") pod \"63830843-4d31-45a7-83c9-9bb2248abe3b\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " Oct 30 02:02:18.327239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327205 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-data\") pod \"6cd04c1d-6bb4-4cd8-99b5-523538908019\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " Oct 30 02:02:18.327239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327240 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-workload-certs\") pod \"6cd04c1d-6bb4-4cd8-99b5-523538908019\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " Oct 30 02:02:18.327797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327276 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-podinfo\") pod \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " Oct 30 02:02:18.327797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327292 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd04c1d-6bb4-4cd8-99b5-523538908019-istiod-ca-cert" pod "6cd04c1d-6bb4-4cd8-99b5-523538908019" (UID: "6cd04c1d-6bb4-4cd8-99b5-523538908019"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:18.327797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327318 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-envoy\") pod \"6cd04c1d-6bb4-4cd8-99b5-523538908019\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " Oct 30 02:02:18.327797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327352 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-workload-socket\") pod \"6cd04c1d-6bb4-4cd8-99b5-523538908019\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " Oct 30 02:02:18.327797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327393 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94053f0f-2fd8-4ab0-914a-256ddadb5d85-kube-api-access-m6x6g\" (UniqueName: \"kubernetes.io/projected/94053f0f-2fd8-4ab0-914a-256ddadb5d85-kube-api-access-m6x6g\") pod \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " Oct 30 02:02:18.327797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327460 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-token\" (UniqueName: \"kubernetes.io/projected/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-token\") pod \"63830843-4d31-45a7-83c9-9bb2248abe3b\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " Oct 30 02:02:18.327797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327554 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-workload-socket\") pod \"63830843-4d31-45a7-83c9-9bb2248abe3b\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " Oct 30 02:02:18.327797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327595 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-podinfo\") pod \"63830843-4d31-45a7-83c9-9bb2248abe3b\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " Oct 30 02:02:18.327797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327654 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-podinfo\") pod \"6cd04c1d-6bb4-4cd8-99b5-523538908019\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " Oct 30 02:02:18.327797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327694 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-token\" (UniqueName: \"kubernetes.io/projected/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-token\") pod \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " Oct 30 02:02:18.327797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327736 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-workload-certs\") pod \"63830843-4d31-45a7-83c9-9bb2248abe3b\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " Oct 30 02:02:18.327797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327765 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cd04c1d-6bb4-4cd8-99b5-523538908019-kube-api-access-jdx77\" (UniqueName: \"kubernetes.io/projected/6cd04c1d-6bb4-4cd8-99b5-523538908019-kube-api-access-jdx77\") pod \"6cd04c1d-6bb4-4cd8-99b5-523538908019\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " Oct 30 02:02:18.328382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327822 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63830843-4d31-45a7-83c9-9bb2248abe3b-kube-api-access-l7wtf\" (UniqueName: \"kubernetes.io/projected/63830843-4d31-45a7-83c9-9bb2248abe3b-kube-api-access-l7wtf\") pod \"63830843-4d31-45a7-83c9-9bb2248abe3b\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " Oct 30 02:02:18.328382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327874 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-credential-socket\") pod \"6cd04c1d-6bb4-4cd8-99b5-523538908019\" (UID: \"6cd04c1d-6bb4-4cd8-99b5-523538908019\") " Oct 30 02:02:18.328382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327905 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istiod-ca-cert\") pod \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\" (UID: \"94053f0f-2fd8-4ab0-914a-256ddadb5d85\") " Oct 30 02:02:18.328382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.327937 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-credential-socket\") pod \"63830843-4d31-45a7-83c9-9bb2248abe3b\" (UID: \"63830843-4d31-45a7-83c9-9bb2248abe3b\") " Oct 30 02:02:18.328714 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.328635 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.328714 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.328660 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd04c1d-6bb4-4cd8-99b5-523538908019-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.328977 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.328918 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-data" pod "63830843-4d31-45a7-83c9-9bb2248abe3b" (UID: "63830843-4d31-45a7-83c9-9bb2248abe3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:18.329142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.329117 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-workload-socket" pod "6cd04c1d-6bb4-4cd8-99b5-523538908019" (UID: "6cd04c1d-6bb4-4cd8-99b5-523538908019"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:18.330632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.329629 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-credential-socket" pod "63830843-4d31-45a7-83c9-9bb2248abe3b" (UID: "63830843-4d31-45a7-83c9-9bb2248abe3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:18.330632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.330065 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63830843-4d31-45a7-83c9-9bb2248abe3b-istiod-ca-cert" pod "63830843-4d31-45a7-83c9-9bb2248abe3b" (UID: "63830843-4d31-45a7-83c9-9bb2248abe3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:18.330632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.330444 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-data" pod "6cd04c1d-6bb4-4cd8-99b5-523538908019" (UID: "6cd04c1d-6bb4-4cd8-99b5-523538908019"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:18.330632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.330601 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-workload-socket" pod "94053f0f-2fd8-4ab0-914a-256ddadb5d85" (UID: "94053f0f-2fd8-4ab0-914a-256ddadb5d85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:18.331585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.330811 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-workload-certs" pod "6cd04c1d-6bb4-4cd8-99b5-523538908019" (UID: "6cd04c1d-6bb4-4cd8-99b5-523538908019"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:18.331585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.331170 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-credential-socket" pod "94053f0f-2fd8-4ab0-914a-256ddadb5d85" (UID: "94053f0f-2fd8-4ab0-914a-256ddadb5d85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:18.331585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.331348 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-data" pod "94053f0f-2fd8-4ab0-914a-256ddadb5d85" (UID: "94053f0f-2fd8-4ab0-914a-256ddadb5d85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:18.331895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.331874 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-podinfo" pod "6cd04c1d-6bb4-4cd8-99b5-523538908019" (UID: "6cd04c1d-6bb4-4cd8-99b5-523538908019"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:02:18.331996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.331936 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-envoy" pod "6cd04c1d-6bb4-4cd8-99b5-523538908019" (UID: "6cd04c1d-6bb4-4cd8-99b5-523538908019"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:18.332081 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.332014 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-token" pod "6cd04c1d-6bb4-4cd8-99b5-523538908019" (UID: "6cd04c1d-6bb4-4cd8-99b5-523538908019"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:18.332892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.332295 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-credential-socket" pod "6cd04c1d-6bb4-4cd8-99b5-523538908019" (UID: "6cd04c1d-6bb4-4cd8-99b5-523538908019"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:18.332892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.332607 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istiod-ca-cert" pod "94053f0f-2fd8-4ab0-914a-256ddadb5d85" (UID: "94053f0f-2fd8-4ab0-914a-256ddadb5d85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:02:18.332892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.333405 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-workload-socket" pod "63830843-4d31-45a7-83c9-9bb2248abe3b" (UID: "63830843-4d31-45a7-83c9-9bb2248abe3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:18.332892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.333458 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-workload-certs" pod "63830843-4d31-45a7-83c9-9bb2248abe3b" (UID: "63830843-4d31-45a7-83c9-9bb2248abe3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:18.332892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.337020 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-envoy" pod "94053f0f-2fd8-4ab0-914a-256ddadb5d85" (UID: "94053f0f-2fd8-4ab0-914a-256ddadb5d85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:18.337912 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.337836 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-token" pod "63830843-4d31-45a7-83c9-9bb2248abe3b" (UID: "63830843-4d31-45a7-83c9-9bb2248abe3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:18.337912 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.337873 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94053f0f-2fd8-4ab0-914a-256ddadb5d85-kube-api-access-m6x6g" pod "94053f0f-2fd8-4ab0-914a-256ddadb5d85" (UID: "94053f0f-2fd8-4ab0-914a-256ddadb5d85"). InnerVolumeSpecName "kube-api-access-m6x6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:18.338074 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.337939 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-podinfo" pod "94053f0f-2fd8-4ab0-914a-256ddadb5d85" (UID: "94053f0f-2fd8-4ab0-914a-256ddadb5d85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:02:18.338325 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.338292 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd04c1d-6bb4-4cd8-99b5-523538908019-kube-api-access-jdx77" pod "6cd04c1d-6bb4-4cd8-99b5-523538908019" (UID: "6cd04c1d-6bb4-4cd8-99b5-523538908019"). InnerVolumeSpecName "kube-api-access-jdx77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:18.338700 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.338632 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-token" pod "94053f0f-2fd8-4ab0-914a-256ddadb5d85" (UID: "94053f0f-2fd8-4ab0-914a-256ddadb5d85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:18.338700 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.338664 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-envoy" pod "63830843-4d31-45a7-83c9-9bb2248abe3b" (UID: "63830843-4d31-45a7-83c9-9bb2248abe3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:02:18.338700 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.338691 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63830843-4d31-45a7-83c9-9bb2248abe3b-kube-api-access-l7wtf" pod "63830843-4d31-45a7-83c9-9bb2248abe3b" (UID: "63830843-4d31-45a7-83c9-9bb2248abe3b"). InnerVolumeSpecName "kube-api-access-l7wtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:02:18.340275 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.340253 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-podinfo" pod "63830843-4d31-45a7-83c9-9bb2248abe3b" (UID: "63830843-4d31-45a7-83c9-9bb2248abe3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:02:18.429736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429652 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7wtf\" (UniqueName: \"kubernetes.io/projected/63830843-4d31-45a7-83c9-9bb2248abe3b-kube-api-access-l7wtf\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.429736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429684 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.429736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429693 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.429736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429702 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.429736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429712 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.429736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429720 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.429736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429727 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.429736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429735 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.429736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429743 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.430364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429751 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.430364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429759 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.430364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429767 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63830843-4d31-45a7-83c9-9bb2248abe3b-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.430364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429774 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.430364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429781 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.430364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429788 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.430364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429797 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.430364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429804 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd04c1d-6bb4-4cd8-99b5-523538908019-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.430364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429811 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6x6g\" (UniqueName: \"kubernetes.io/projected/94053f0f-2fd8-4ab0-914a-256ddadb5d85-kube-api-access-m6x6g\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.430364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429819 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.430364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429827 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.430364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429835 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63830843-4d31-45a7-83c9-9bb2248abe3b-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.430364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429842 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd04c1d-6bb4-4cd8-99b5-523538908019-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.430364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429849 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94053f0f-2fd8-4ab0-914a-256ddadb5d85-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.430364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429857 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63830843-4d31-45a7-83c9-9bb2248abe3b-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.430364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.429866 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdx77\" (UniqueName: \"kubernetes.io/projected/6cd04c1d-6bb4-4cd8-99b5-523538908019-kube-api-access-jdx77\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:02:18.727947 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.727839 5998 generic.go:356] "Generic (PLEG): container finished" podID="94053f0f-2fd8-4ab0-914a-256ddadb5d85" containerID="3027f63794dc2178391b57d4f2c91a0592db3cf295ea9b87658997a8e772778b" exitCode=0 Oct 30 02:02:18.727947 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.727900 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" Oct 30 02:02:18.727947 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.727931 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" event={"ID":"94053f0f-2fd8-4ab0-914a-256ddadb5d85","Type":"ContainerDied","Data":"3027f63794dc2178391b57d4f2c91a0592db3cf295ea9b87658997a8e772778b"} Oct 30 02:02:18.728292 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.727994 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk" event={"ID":"94053f0f-2fd8-4ab0-914a-256ddadb5d85","Type":"ContainerDied","Data":"3a6c94ab901a1f5f028b5dcd2358dc1fa1fd39ad31bdde2ad89f3fe51ee23676"} Oct 30 02:02:18.728292 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.728047 5998 scope.go:122] "RemoveContainer" containerID="3027f63794dc2178391b57d4f2c91a0592db3cf295ea9b87658997a8e772778b" Oct 30 02:02:18.730045 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.730020 5998 generic.go:356] "Generic (PLEG): container finished" podID="6cd04c1d-6bb4-4cd8-99b5-523538908019" containerID="a910a7c81dc5da24505757cd5dabbad04e5a5cdd19a3122057bdef721cae0c85" exitCode=0 Oct 30 02:02:18.730190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.730080 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" Oct 30 02:02:18.730190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.730165 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" event={"ID":"6cd04c1d-6bb4-4cd8-99b5-523538908019","Type":"ContainerDied","Data":"a910a7c81dc5da24505757cd5dabbad04e5a5cdd19a3122057bdef721cae0c85"} Oct 30 02:02:18.730308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.730206 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5" event={"ID":"6cd04c1d-6bb4-4cd8-99b5-523538908019","Type":"ContainerDied","Data":"fe3ceb4da20823f6885f60cdfbcdcfdb52caf7c3d41f2f15efe745ab7ccb0c3a"} Oct 30 02:02:18.731924 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.731902 5998 generic.go:356] "Generic (PLEG): container finished" podID="63830843-4d31-45a7-83c9-9bb2248abe3b" containerID="0f2bd476b2115992a343a857c0baab5dbec6e4121a745c82ad3e0e24a24db26e" exitCode=0 Oct 30 02:02:18.732052 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.731972 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" Oct 30 02:02:18.732052 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.732025 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" event={"ID":"63830843-4d31-45a7-83c9-9bb2248abe3b","Type":"ContainerDied","Data":"0f2bd476b2115992a343a857c0baab5dbec6e4121a745c82ad3e0e24a24db26e"} Oct 30 02:02:18.732193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.732127 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq" event={"ID":"63830843-4d31-45a7-83c9-9bb2248abe3b","Type":"ContainerDied","Data":"c3589b335c2f0748c19ef15df2c99dd3e03c09806aa654457efdbdf9cbdef5d6"} Oct 30 02:02:18.739958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.739942 5998 scope.go:122] "RemoveContainer" containerID="3027f63794dc2178391b57d4f2c91a0592db3cf295ea9b87658997a8e772778b" Oct 30 02:02:18.740282 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:02:18.740252 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3027f63794dc2178391b57d4f2c91a0592db3cf295ea9b87658997a8e772778b\": container with ID starting with 3027f63794dc2178391b57d4f2c91a0592db3cf295ea9b87658997a8e772778b not found: ID does not exist" containerID="3027f63794dc2178391b57d4f2c91a0592db3cf295ea9b87658997a8e772778b" Oct 30 02:02:18.740373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.740284 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3027f63794dc2178391b57d4f2c91a0592db3cf295ea9b87658997a8e772778b"} err="failed to get container status \"3027f63794dc2178391b57d4f2c91a0592db3cf295ea9b87658997a8e772778b\": rpc error: code = NotFound desc = could not find container \"3027f63794dc2178391b57d4f2c91a0592db3cf295ea9b87658997a8e772778b\": container with ID starting with 3027f63794dc2178391b57d4f2c91a0592db3cf295ea9b87658997a8e772778b not found: ID does not exist" Oct 30 02:02:18.740373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.740305 5998 scope.go:122] "RemoveContainer" containerID="a910a7c81dc5da24505757cd5dabbad04e5a5cdd19a3122057bdef721cae0c85" Oct 30 02:02:18.750034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.750018 5998 scope.go:122] "RemoveContainer" containerID="a910a7c81dc5da24505757cd5dabbad04e5a5cdd19a3122057bdef721cae0c85" Oct 30 02:02:18.750400 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:02:18.750367 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a910a7c81dc5da24505757cd5dabbad04e5a5cdd19a3122057bdef721cae0c85\": container with ID starting with a910a7c81dc5da24505757cd5dabbad04e5a5cdd19a3122057bdef721cae0c85 not found: ID does not exist" containerID="a910a7c81dc5da24505757cd5dabbad04e5a5cdd19a3122057bdef721cae0c85" Oct 30 02:02:18.750512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.750409 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a910a7c81dc5da24505757cd5dabbad04e5a5cdd19a3122057bdef721cae0c85"} err="failed to get container status \"a910a7c81dc5da24505757cd5dabbad04e5a5cdd19a3122057bdef721cae0c85\": rpc error: code = NotFound desc = could not find container \"a910a7c81dc5da24505757cd5dabbad04e5a5cdd19a3122057bdef721cae0c85\": container with ID starting with a910a7c81dc5da24505757cd5dabbad04e5a5cdd19a3122057bdef721cae0c85 not found: ID does not exist" Oct 30 02:02:18.750512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.750433 5998 scope.go:122] "RemoveContainer" containerID="0f2bd476b2115992a343a857c0baab5dbec6e4121a745c82ad3e0e24a24db26e" Oct 30 02:02:18.761566 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.761493 5998 scope.go:122] "RemoveContainer" containerID="0f2bd476b2115992a343a857c0baab5dbec6e4121a745c82ad3e0e24a24db26e" Oct 30 02:02:18.762674 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:02:18.762507 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f2bd476b2115992a343a857c0baab5dbec6e4121a745c82ad3e0e24a24db26e\": container with ID starting with 0f2bd476b2115992a343a857c0baab5dbec6e4121a745c82ad3e0e24a24db26e not found: ID does not exist" containerID="0f2bd476b2115992a343a857c0baab5dbec6e4121a745c82ad3e0e24a24db26e" Oct 30 02:02:18.762674 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.762542 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2bd476b2115992a343a857c0baab5dbec6e4121a745c82ad3e0e24a24db26e"} err="failed to get container status \"0f2bd476b2115992a343a857c0baab5dbec6e4121a745c82ad3e0e24a24db26e\": rpc error: code = NotFound desc = could not find container \"0f2bd476b2115992a343a857c0baab5dbec6e4121a745c82ad3e0e24a24db26e\": container with ID starting with 0f2bd476b2115992a343a857c0baab5dbec6e4121a745c82ad3e0e24a24db26e not found: ID does not exist" Oct 30 02:02:18.779745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.779715 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk"] Oct 30 02:02:18.791072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.791049 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-zghmk"] Oct 30 02:02:18.820613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.820578 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq"] Oct 30 02:02:18.825488 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.825457 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-kf6tq"] Oct 30 02:02:18.857082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.857052 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5"] Oct 30 02:02:18.860860 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:18.860834 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-sdrx5"] Oct 30 02:02:19.097659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:19.097577 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:20.541297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:20.541264 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63830843-4d31-45a7-83c9-9bb2248abe3b" path="/var/lib/kubelet/pods/63830843-4d31-45a7-83c9-9bb2248abe3b/volumes" Oct 30 02:02:20.541999 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:20.541976 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cd04c1d-6bb4-4cd8-99b5-523538908019" path="/var/lib/kubelet/pods/6cd04c1d-6bb4-4cd8-99b5-523538908019/volumes" Oct 30 02:02:20.542567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:20.542545 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94053f0f-2fd8-4ab0-914a-256ddadb5d85" path="/var/lib/kubelet/pods/94053f0f-2fd8-4ab0-914a-256ddadb5d85/volumes" Oct 30 02:02:20.564405 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:20.564370 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:20.673836 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:20.673779 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:21.264404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:21.264371 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:21.381777 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:21.381726 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:21.731026 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:21.730995 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:22.548570 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:22.548537 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:23.135177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:23.135071 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:23.354887 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:23.354859 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:23.465897 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:23.465860 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:24.736957 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:24.736914 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:24.848562 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:24.848518 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:24.959518 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:24.959484 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:25.225494 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:02:25.225455 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:02:26.477535 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:26.477493 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:28.147465 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:28.147422 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:29.243588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:29.243552 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:29.852307 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:29.852274 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:29.964111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:29.964066 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:30.222635 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:30.222601 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:31.676936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:31.676907 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:31.927082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:31.926999 5998 scope.go:122] "RemoveContainer" containerID="9890fbe8a63995ce74d5ee9f7b1147960f47f0579c4154f77fb7675706719b0c" Oct 30 02:02:31.936998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:31.936971 5998 scope.go:122] "RemoveContainer" containerID="e66d0c11813d8941b2eda296521bd3e2f20f756061abd3b9e638f82c95846791" Oct 30 02:02:31.946126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:31.946074 5998 scope.go:122] "RemoveContainer" containerID="4d82dee480d17b4dde5967551ab2f2317662d073d8c1153e9e9d85153476eae6" Oct 30 02:02:32.829713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:32.829681 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:02:33.344488 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:33.344451 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:35.338794 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:35.338757 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:35.473018 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:35.472979 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:35.716192 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:35.716153 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:36.930573 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:36.930531 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:37.106533 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:37.105446 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:37.216191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:37.216072 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:37.711119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:37.711063 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:39.299711 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:39.299676 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:40.867848 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:40.867813 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:43.092557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:43.092467 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:44.688800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:44.688763 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:45.895888 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:45.895855 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:46.010893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:46.010855 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:46.294147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:46.294072 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:46.692157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:46.692119 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:47.169934 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:47.169892 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:48.474232 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:48.474198 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:48.665620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:48.665588 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:49.114673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.114630 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td"] Oct 30 02:02:49.115647 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115610 5998 cpu_manager.go:414] "containerMap: removing container" podUID="4f80e088-78c9-4b95-98cc-4d1dd67d9a34" containerName="payload-pre-processing" Oct 30 02:02:49.115647 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115641 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f80e088-78c9-4b95-98cc-4d1dd67d9a34" containerName="payload-pre-processing" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115661 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" containerName="istio-proxy" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115669 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" containerName="istio-proxy" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115679 5998 cpu_manager.go:414] "containerMap: removing container" podUID="af80b61b-f6eb-4e60-abbd-83495acec7d9" containerName="cleanup" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115687 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af80b61b-f6eb-4e60-abbd-83495acec7d9" containerName="cleanup" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115700 5998 cpu_manager.go:414] "containerMap: removing container" podUID="847d96ae-df65-41f4-8663-53fdbd7a4b97" containerName="cleanup" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115707 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847d96ae-df65-41f4-8663-53fdbd7a4b97" containerName="cleanup" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115717 5998 cpu_manager.go:414] "containerMap: removing container" podUID="48178a43-c541-48df-a3cd-5712a41998e9" containerName="cleanup" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115723 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48178a43-c541-48df-a3cd-5712a41998e9" containerName="cleanup" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115735 5998 cpu_manager.go:414] "containerMap: removing container" podUID="847d96ae-df65-41f4-8663-53fdbd7a4b97" containerName="cleanup" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115742 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847d96ae-df65-41f4-8663-53fdbd7a4b97" containerName="cleanup" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115754 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ffb61d79-7ce6-46d2-9f0d-712fb8a446a1" containerName="cleanup" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115761 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb61d79-7ce6-46d2-9f0d-712fb8a446a1" containerName="cleanup" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115774 5998 cpu_manager.go:414] "containerMap: removing container" podUID="63830843-4d31-45a7-83c9-9bb2248abe3b" containerName="istio-proxy" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115782 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63830843-4d31-45a7-83c9-9bb2248abe3b" containerName="istio-proxy" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115796 5998 cpu_manager.go:414] "containerMap: removing container" podUID="46590fa3-d39f-4df5-8c5b-96520faa7625" containerName="istio-proxy" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115803 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46590fa3-d39f-4df5-8c5b-96520faa7625" containerName="istio-proxy" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115817 5998 cpu_manager.go:414] "containerMap: removing container" podUID="847d96ae-df65-41f4-8663-53fdbd7a4b97" containerName="cleanup" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115824 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="847d96ae-df65-41f4-8663-53fdbd7a4b97" containerName="cleanup" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115839 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a6132fae-af61-4920-b993-8eb4760609ff" containerName="cleanup" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115846 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6132fae-af61-4920-b993-8eb4760609ff" containerName="cleanup" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115855 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1c0254eb-8332-4172-b1ba-a8235aa882a8" containerName="cleanup" Oct 30 02:02:49.115847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115862 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0254eb-8332-4172-b1ba-a8235aa882a8" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115879 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3605446f-2664-4b96-9a9f-e13aab64faee" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115885 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3605446f-2664-4b96-9a9f-e13aab64faee" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115901 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a6132fae-af61-4920-b993-8eb4760609ff" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115908 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6132fae-af61-4920-b993-8eb4760609ff" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115917 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ffb61d79-7ce6-46d2-9f0d-712fb8a446a1" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115923 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb61d79-7ce6-46d2-9f0d-712fb8a446a1" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.115991 5998 cpu_manager.go:414] "containerMap: removing container" podUID="5c72e866-870f-4f28-8e32-eb303f84488d" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116002 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c72e866-870f-4f28-8e32-eb303f84488d" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116011 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1c0254eb-8332-4172-b1ba-a8235aa882a8" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116019 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0254eb-8332-4172-b1ba-a8235aa882a8" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116037 5998 cpu_manager.go:414] "containerMap: removing container" podUID="9901c539-93e6-407d-b99c-b7cd2ad64356" containerName="payload-pre-processing" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116044 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9901c539-93e6-407d-b99c-b7cd2ad64356" containerName="payload-pre-processing" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116056 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ffb61d79-7ce6-46d2-9f0d-712fb8a446a1" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116064 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb61d79-7ce6-46d2-9f0d-712fb8a446a1" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116077 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116083 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116109 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1c0254eb-8332-4172-b1ba-a8235aa882a8" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116115 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0254eb-8332-4172-b1ba-a8235aa882a8" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116125 5998 cpu_manager.go:414] "containerMap: removing container" podUID="af80b61b-f6eb-4e60-abbd-83495acec7d9" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116132 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af80b61b-f6eb-4e60-abbd-83495acec7d9" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116144 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ee325398-ca84-48b8-90b1-105e16fbebd4" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116151 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee325398-ca84-48b8-90b1-105e16fbebd4" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116163 5998 cpu_manager.go:414] "containerMap: removing container" podUID="6cd04c1d-6bb4-4cd8-99b5-523538908019" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116170 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd04c1d-6bb4-4cd8-99b5-523538908019" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116179 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1008ca2d-680d-496b-acec-7ac087a0d48c" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116185 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1008ca2d-680d-496b-acec-7ac087a0d48c" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116199 5998 cpu_manager.go:414] "containerMap: removing container" podUID="94053f0f-2fd8-4ab0-914a-256ddadb5d85" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116206 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94053f0f-2fd8-4ab0-914a-256ddadb5d85" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116217 5998 cpu_manager.go:414] "containerMap: removing container" podUID="48178a43-c541-48df-a3cd-5712a41998e9" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116224 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48178a43-c541-48df-a3cd-5712a41998e9" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116236 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a6132fae-af61-4920-b993-8eb4760609ff" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116244 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6132fae-af61-4920-b993-8eb4760609ff" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116255 5998 cpu_manager.go:414] "containerMap: removing container" podUID="48178a43-c541-48df-a3cd-5712a41998e9" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116261 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48178a43-c541-48df-a3cd-5712a41998e9" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116271 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1008ca2d-680d-496b-acec-7ac087a0d48c" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116278 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1008ca2d-680d-496b-acec-7ac087a0d48c" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116292 5998 cpu_manager.go:414] "containerMap: removing container" podUID="0b6f67a0-2dae-400a-abd7-7c65fbdac225" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116299 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6f67a0-2dae-400a-abd7-7c65fbdac225" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116308 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1008ca2d-680d-496b-acec-7ac087a0d48c" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116314 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1008ca2d-680d-496b-acec-7ac087a0d48c" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116327 5998 cpu_manager.go:414] "containerMap: removing container" podUID="19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116334 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116344 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ee325398-ca84-48b8-90b1-105e16fbebd4" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116351 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee325398-ca84-48b8-90b1-105e16fbebd4" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116369 5998 cpu_manager.go:414] "containerMap: removing container" podUID="5c72e866-870f-4f28-8e32-eb303f84488d" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116375 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c72e866-870f-4f28-8e32-eb303f84488d" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116388 5998 cpu_manager.go:414] "containerMap: removing container" podUID="323ecfe9-7182-4e29-9774-dd3dd728fc09" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116396 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323ecfe9-7182-4e29-9774-dd3dd728fc09" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116409 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3daf0043-f037-417f-bf48-7a8f53565395" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116415 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3daf0043-f037-417f-bf48-7a8f53565395" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116429 5998 cpu_manager.go:414] "containerMap: removing container" podUID="323ecfe9-7182-4e29-9774-dd3dd728fc09" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116436 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323ecfe9-7182-4e29-9774-dd3dd728fc09" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116574 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="9901c539-93e6-407d-b99c-b7cd2ad64356" containerName="payload-pre-processing" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116584 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb61d79-7ce6-46d2-9f0d-712fb8a446a1" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116595 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c105b9-9c67-48ab-821c-b16c9f9ef5e5" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116603 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0254eb-8332-4172-b1ba-a8235aa882a8" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116612 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="af80b61b-f6eb-4e60-abbd-83495acec7d9" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116619 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee325398-ca84-48b8-90b1-105e16fbebd4" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116628 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd04c1d-6bb4-4cd8-99b5-523538908019" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116636 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="af80b61b-f6eb-4e60-abbd-83495acec7d9" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116646 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1008ca2d-680d-496b-acec-7ac087a0d48c" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116656 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="94053f0f-2fd8-4ab0-914a-256ddadb5d85" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116663 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="48178a43-c541-48df-a3cd-5712a41998e9" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116671 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6132fae-af61-4920-b993-8eb4760609ff" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116681 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="48178a43-c541-48df-a3cd-5712a41998e9" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116688 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1008ca2d-680d-496b-acec-7ac087a0d48c" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116696 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6f67a0-2dae-400a-abd7-7c65fbdac225" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116703 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="19bebf03-9dfe-4acf-a5e7-b4e8e6ec65f8" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116713 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c72e866-870f-4f28-8e32-eb303f84488d" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116722 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee325398-ca84-48b8-90b1-105e16fbebd4" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116729 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="323ecfe9-7182-4e29-9774-dd3dd728fc09" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116740 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3daf0043-f037-417f-bf48-7a8f53565395" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116750 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="323ecfe9-7182-4e29-9774-dd3dd728fc09" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116759 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f80e088-78c9-4b95-98cc-4d1dd67d9a34" containerName="payload-pre-processing" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116768 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bbcaa62-d63a-4bf4-ae72-8b091238cc2f" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116778 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="847d96ae-df65-41f4-8663-53fdbd7a4b97" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116787 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="847d96ae-df65-41f4-8663-53fdbd7a4b97" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116795 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="63830843-4d31-45a7-83c9-9bb2248abe3b" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116804 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="46590fa3-d39f-4df5-8c5b-96520faa7625" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116813 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="847d96ae-df65-41f4-8663-53fdbd7a4b97" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116822 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6132fae-af61-4920-b993-8eb4760609ff" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116829 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0254eb-8332-4172-b1ba-a8235aa882a8" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116839 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3605446f-2664-4b96-9a9f-e13aab64faee" containerName="istio-proxy" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116852 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6132fae-af61-4920-b993-8eb4760609ff" containerName="cleanup" Oct 30 02:02:49.117036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.116861 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb61d79-7ce6-46d2-9f0d-712fb8a446a1" containerName="cleanup" Oct 30 02:02:49.123235 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.123210 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.141207 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.141176 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-2979be-openshift-default-dockercfg-jmq5l\"" Oct 30 02:02:49.156767 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.156732 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td"] Oct 30 02:02:49.173913 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.173879 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:49.197437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.197406 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-data\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.197597 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.197449 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-workload-certs\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.197597 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.197476 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-envoy\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.197597 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.197525 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-token\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.197728 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.197637 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istiod-ca-cert\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.197728 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.197675 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-credential-socket\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.197808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.197768 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6grrz\" (UniqueName: \"kubernetes.io/projected/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-kube-api-access-6grrz\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.197808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.197802 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-workload-socket\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.197917 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.197904 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-podinfo\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.306933 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.306885 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grrz\" (UniqueName: \"kubernetes.io/projected/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-kube-api-access-6grrz\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.307700 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.307674 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-workload-socket\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.308383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.308363 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-podinfo\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.309076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.309058 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-data\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.309543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.309525 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-workload-certs\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.309676 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.309659 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-envoy\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.309779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.309766 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-token\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.310523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.310194 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-workload-certs\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.310523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.308260 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-workload-socket\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.310523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.309465 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-data\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.311054 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.311030 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istiod-ca-cert\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.311255 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.311238 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-credential-socket\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.311729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.311708 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-credential-socket\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.318122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.311788 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istiod-ca-cert\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.318807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.318744 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-podinfo\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.343975 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.343909 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-envoy\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.367227 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.367151 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6grrz\" (UniqueName: \"kubernetes.io/projected/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-kube-api-access-6grrz\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.377109 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.368186 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-token\") pod \"e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.446172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.446129 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:49.471062 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.468341 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:49.794904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.794868 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td"] Oct 30 02:02:49.798588 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:02:49.798553 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad87fa4_78c9_4e60_8706_a78dbd1002b1.slice/crio-0d316c49ed3057d9a207ac016c55f2d26d6a7af2dbf7b00abf071bb5e77c0351 WatchSource:0}: Error finding container 0d316c49ed3057d9a207ac016c55f2d26d6a7af2dbf7b00abf071bb5e77c0351: Status 404 returned error can't find the container with id 0d316c49ed3057d9a207ac016c55f2d26d6a7af2dbf7b00abf071bb5e77c0351 Oct 30 02:02:49.801380 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.801347 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:02:49.801498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.801427 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:02:49.801498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.801477 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:02:49.988558 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.988520 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" event={"ID":"0ad87fa4-78c9-4e60-8706-a78dbd1002b1","Type":"ContainerStarted","Data":"e806753998881e2282853e62697af3fffc9de7171f25c01c8f0f3d17b6c1fc08"} Oct 30 02:02:49.988751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:49.988584 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" event={"ID":"0ad87fa4-78c9-4e60-8706-a78dbd1002b1","Type":"ContainerStarted","Data":"0d316c49ed3057d9a207ac016c55f2d26d6a7af2dbf7b00abf071bb5e77c0351"} Oct 30 02:02:50.046157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:50.045104 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" podStartSLOduration=1.045067299 podStartE2EDuration="1.045067299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:02:50.033042734 +0000 UTC m=+1999.611833815" watchObservedRunningTime="2026-10-30 02:02:50.045067299 +0000 UTC m=+1999.623858385" Oct 30 02:02:50.270965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:50.270929 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:50.446674 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:50.446638 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:50.453605 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:50.453578 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:50.711945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:50.711855 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:50.833507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:50.833463 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:50.994792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:50.994708 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:50.995824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:50.995803 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:02:52.157963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:52.157924 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:52.438907 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:52.438869 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:52.560338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:52.560306 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:53.960598 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:53.960559 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:54.904321 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:54.904274 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:55.015185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:55.015147 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:55.329685 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:02:55.329648 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:02:55.813997 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:55.813956 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:58.118666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:58.118633 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:58.413291 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:58.413255 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:02:59.573236 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:02:59.573191 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:00.077324 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:00.077285 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:00.290083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:00.290050 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:00.665447 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:00.665407 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:02.185237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:02.185207 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:03.624289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:03.624255 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:03.911596 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:03.911562 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:03.946613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:03.946580 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n"] Oct 30 02:03:03.947107 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:03.947078 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ee325398-ca84-48b8-90b1-105e16fbebd4" containerName="cleanup" Oct 30 02:03:03.947168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:03.947108 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee325398-ca84-48b8-90b1-105e16fbebd4" containerName="cleanup" Oct 30 02:03:03.947168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:03.947152 5998 cpu_manager.go:414] "containerMap: removing container" podUID="af80b61b-f6eb-4e60-abbd-83495acec7d9" containerName="cleanup" Oct 30 02:03:03.947168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:03.947157 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af80b61b-f6eb-4e60-abbd-83495acec7d9" containerName="cleanup" Oct 30 02:03:03.947281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:03.947245 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="48178a43-c541-48df-a3cd-5712a41998e9" containerName="cleanup" Oct 30 02:03:03.947281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:03.947256 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1008ca2d-680d-496b-acec-7ac087a0d48c" containerName="cleanup" Oct 30 02:03:03.947281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:03.947265 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c72e866-870f-4f28-8e32-eb303f84488d" containerName="istio-proxy" Oct 30 02:03:03.947281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:03.947271 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee325398-ca84-48b8-90b1-105e16fbebd4" containerName="cleanup" Oct 30 02:03:03.947410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:03.947283 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb61d79-7ce6-46d2-9f0d-712fb8a446a1" containerName="cleanup" Oct 30 02:03:03.947410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:03.947290 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0254eb-8332-4172-b1ba-a8235aa882a8" containerName="cleanup" Oct 30 02:03:03.947410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:03.947298 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="af80b61b-f6eb-4e60-abbd-83495acec7d9" containerName="cleanup" Oct 30 02:03:03.950814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:03.950790 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:03.967386 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:03.966528 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-70a069-openshift-default-dockercfg-2bzd9\"" Oct 30 02:03:03.983310 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:03.983279 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n"] Oct 30 02:03:04.056773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.056730 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hpxw\" (UniqueName: \"kubernetes.io/projected/41302545-fcf3-4f12-80b1-14d1992bedf5-kube-api-access-4hpxw\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.056773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.056768 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-envoy\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.057059 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.056801 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-credential-socket\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.057059 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.056926 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-data\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.057203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.057078 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-token\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.057203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.057195 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-podinfo\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.057320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.057223 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41302545-fcf3-4f12-80b1-14d1992bedf5-istiod-ca-cert\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.057320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.057257 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-workload-socket\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.057320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.057314 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-workload-certs\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.158593 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.158546 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-podinfo\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.158593 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.158595 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41302545-fcf3-4f12-80b1-14d1992bedf5-istiod-ca-cert\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.158853 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.158622 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-workload-socket\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.158853 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.158646 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-workload-certs\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.158853 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.158679 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hpxw\" (UniqueName: \"kubernetes.io/projected/41302545-fcf3-4f12-80b1-14d1992bedf5-kube-api-access-4hpxw\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.158853 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.158696 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-envoy\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.158853 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.158741 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-credential-socket\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.158853 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.158786 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-data\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.159199 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.158882 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-token\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.159199 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.159141 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-workload-certs\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.159321 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.159202 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-workload-socket\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.159536 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.159509 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-data\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.159669 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.159533 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41302545-fcf3-4f12-80b1-14d1992bedf5-istiod-ca-cert\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.159669 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.159557 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-credential-socket\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.161113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.161076 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-envoy\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.161260 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.161240 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-podinfo\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.181643 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.181574 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-token\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.181876 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.181853 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hpxw\" (UniqueName: \"kubernetes.io/projected/41302545-fcf3-4f12-80b1-14d1992bedf5-kube-api-access-4hpxw\") pod \"e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.279157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.279083 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:04.604463 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.604367 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n"] Oct 30 02:03:04.608484 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:03:04.608443 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41302545_fcf3_4f12_80b1_14d1992bedf5.slice/crio-bc2a587658a811036dbd666b7a95a10169cb5e6cdecdcb8b34cbe638d93fe114 WatchSource:0}: Error finding container bc2a587658a811036dbd666b7a95a10169cb5e6cdecdcb8b34cbe638d93fe114: Status 404 returned error can't find the container with id bc2a587658a811036dbd666b7a95a10169cb5e6cdecdcb8b34cbe638d93fe114 Oct 30 02:03:04.611650 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.611610 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:03:04.611749 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.611701 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:03:04.611789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:04.611748 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:03:05.104891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:05.104769 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" event={"ID":"41302545-fcf3-4f12-80b1-14d1992bedf5","Type":"ContainerStarted","Data":"a6b7c49602c455bafdd28b55ddc97d0e9e274726f3d236f68a424b26cff1f24a"} Oct 30 02:03:05.105532 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:05.104859 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" event={"ID":"41302545-fcf3-4f12-80b1-14d1992bedf5","Type":"ContainerStarted","Data":"bc2a587658a811036dbd666b7a95a10169cb5e6cdecdcb8b34cbe638d93fe114"} Oct 30 02:03:05.189297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:05.189244 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" podStartSLOduration=2.189227346 podStartE2EDuration="2.189227346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:03:05.175059581 +0000 UTC m=+2014.753850676" watchObservedRunningTime="2026-10-30 02:03:05.189227346 +0000 UTC m=+2014.768018426" Oct 30 02:03:05.279564 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:05.279533 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:05.281243 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:05.281211 5998 patch_prober.go:29] interesting pod/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 02:03:05.281399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:05.281274 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" podUID="41302545-fcf3-4f12-80b1-14d1992bedf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 02:03:06.197556 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:06.197515 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:06.279585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:06.279534 5998 patch_prober.go:29] interesting pod/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 02:03:06.279762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:06.279597 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" podUID="41302545-fcf3-4f12-80b1-14d1992bedf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 02:03:07.279523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:07.279478 5998 patch_prober.go:29] interesting pod/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 02:03:07.280053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:07.279548 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" podUID="41302545-fcf3-4f12-80b1-14d1992bedf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 02:03:08.036629 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:08.036573 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:08.280268 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:08.280236 5998 patch_prober.go:29] interesting pod/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 02:03:08.280715 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:08.280287 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" podUID="41302545-fcf3-4f12-80b1-14d1992bedf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 02:03:09.280106 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:09.280066 5998 patch_prober.go:29] interesting pod/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 02:03:09.280383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:09.280141 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" podUID="41302545-fcf3-4f12-80b1-14d1992bedf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 02:03:09.967026 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:09.966992 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:10.280561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:10.280463 5998 patch_prober.go:29] interesting pod/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 02:03:10.281193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:10.281158 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" podUID="41302545-fcf3-4f12-80b1-14d1992bedf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 02:03:11.284757 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:11.284725 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:11.285204 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:11.285127 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:11.285788 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:11.285768 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:03:11.288076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:11.288053 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:11.470709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:11.455555 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:12.012469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:12.012386 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:12.033186 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:12.033148 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs"] Oct 30 02:03:12.037499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:12.037462 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs" Oct 30 02:03:12.068745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:12.068711 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs"] Oct 30 02:03:12.086603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:12.086577 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-168989\"" Oct 30 02:03:12.089199 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:12.089178 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-168989-dockercfg-cqrn5\"" Oct 30 02:03:12.137491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:12.137452 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3049cb-dd1f-484c-addf-a9cf4285da41-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs\" (UID: \"eb3049cb-dd1f-484c-addf-a9cf4285da41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs" Oct 30 02:03:12.137695 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:12.137592 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf2nm\" (UniqueName: \"kubernetes.io/projected/eb3049cb-dd1f-484c-addf-a9cf4285da41-kube-api-access-cf2nm\") pod \"payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs\" (UID: \"eb3049cb-dd1f-484c-addf-a9cf4285da41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs" Oct 30 02:03:12.239400 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:12.239348 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3049cb-dd1f-484c-addf-a9cf4285da41-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs\" (UID: \"eb3049cb-dd1f-484c-addf-a9cf4285da41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs" Oct 30 02:03:12.239609 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:12.239559 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf2nm\" (UniqueName: \"kubernetes.io/projected/eb3049cb-dd1f-484c-addf-a9cf4285da41-kube-api-access-cf2nm\") pod \"payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs\" (UID: \"eb3049cb-dd1f-484c-addf-a9cf4285da41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs" Oct 30 02:03:12.240196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:12.240170 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3049cb-dd1f-484c-addf-a9cf4285da41-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs\" (UID: \"eb3049cb-dd1f-484c-addf-a9cf4285da41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs" Oct 30 02:03:12.262863 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:12.262798 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf2nm\" (UniqueName: \"kubernetes.io/projected/eb3049cb-dd1f-484c-addf-a9cf4285da41-kube-api-access-cf2nm\") pod \"payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs\" (UID: \"eb3049cb-dd1f-484c-addf-a9cf4285da41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs" Oct 30 02:03:12.352300 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:12.352262 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs" Oct 30 02:03:12.626727 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:12.626511 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:12.670574 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:12.667809 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs"] Oct 30 02:03:12.921458 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:12.921415 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:13.192564 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:13.192465 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs" event={"ID":"eb3049cb-dd1f-484c-addf-a9cf4285da41","Type":"ContainerStarted","Data":"7e7806268237253627fe8ef6a16a5118853f30246c177f80e0551f0750266286"} Oct 30 02:03:13.192564 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:13.192560 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs" event={"ID":"eb3049cb-dd1f-484c-addf-a9cf4285da41","Type":"ContainerStarted","Data":"4832267aa7ca2b2b91e3aa7f5fa8fc4fdc07c586d9a9492ba3c83becff0dddd3"} Oct 30 02:03:13.192795 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:13.192634 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs" Oct 30 02:03:13.237324 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:13.237264 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs" podStartSLOduration=1.237244784 podStartE2EDuration="1.237244784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:03:13.236257952 +0000 UTC m=+2022.815049035" watchObservedRunningTime="2026-10-30 02:03:13.237244784 +0000 UTC m=+2022.816035871" Oct 30 02:03:14.191898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:14.191859 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:14.331364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:14.331327 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:14.711852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:14.711816 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:15.446542 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:15.446503 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:15.868746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:15.868651 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:15.979276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:15.979238 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:16.638933 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:16.638890 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:16.759289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:16.759232 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:16.946721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:16.946680 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:18.389857 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:18.389821 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:19.407476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:19.407440 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:03:20.023649 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:20.023613 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:21.720256 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:21.720221 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:21.993969 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:21.993805 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:22.119528 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:22.119424 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:22.649855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:22.649819 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:24.077872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:24.077832 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:24.200637 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:24.200603 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs" Oct 30 02:03:25.223491 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:03:25.223452 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:03:25.652542 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:25.652458 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:27.382798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:27.382765 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:27.771426 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:27.771394 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:28.278654 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:28.278596 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:28.563793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:28.563605 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:29.292247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:29.292211 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:29.660322 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:29.660238 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:31.384924 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:31.384837 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:32.003901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:32.003869 5998 scope.go:122] "RemoveContainer" containerID="90120c19222e8cc32b23ff74e79305442dfe53f930aba61cffa17e905c38b44e" Oct 30 02:03:32.017168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:32.017144 5998 scope.go:122] "RemoveContainer" containerID="c9e02ad349dfad795f7407adf6adad3edaf5c94b744189b87a0c919142e07c36" Oct 30 02:03:32.387517 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:32.387440 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:32.501144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:32.501109 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:33.415340 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:33.415284 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:34.188061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:34.188016 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:34.297529 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:34.297497 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:35.629179 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:35.629142 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:35.861792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:35.861754 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:37.506210 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:37.506155 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:38.354255 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:38.354227 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:38.469904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:38.469842 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:38.604116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:38.600290 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:40.195111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:40.195055 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:41.769527 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:41.769490 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:43.527071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:43.527027 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:44.488494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:44.488458 5998 generic.go:356] "Generic (PLEG): container finished" podID="0ad87fa4-78c9-4e60-8706-a78dbd1002b1" containerID="e806753998881e2282853e62697af3fffc9de7171f25c01c8f0f3d17b6c1fc08" exitCode=0 Oct 30 02:03:44.488708 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:44.488525 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" event={"ID":"0ad87fa4-78c9-4e60-8706-a78dbd1002b1","Type":"ContainerDied","Data":"e806753998881e2282853e62697af3fffc9de7171f25c01c8f0f3d17b6c1fc08"} Oct 30 02:03:44.488939 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:44.488925 5998 scope.go:122] "RemoveContainer" containerID="e806753998881e2282853e62697af3fffc9de7171f25c01c8f0f3d17b6c1fc08" Oct 30 02:03:44.490510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:44.490482 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:03:44.490580 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:44.490552 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:03:44.490616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:44.490587 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:03:45.270554 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:45.270510 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:45.496176 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:45.496140 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" event={"ID":"0ad87fa4-78c9-4e60-8706-a78dbd1002b1","Type":"ContainerStarted","Data":"0bcabb375dd5f7274f7f3a9fe236f7f6e60d108a3e1e7c73b1e7e676110e270d"} Oct 30 02:03:46.447365 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:46.447163 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:03:46.448822 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:46.448481 5998 patch_prober.go:29] interesting pod/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 02:03:46.448822 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:46.448539 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" podUID="0ad87fa4-78c9-4e60-8706-a78dbd1002b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 02:03:46.638350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:46.638273 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:46.760763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:46.760683 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:46.892400 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:46.892353 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:47.454469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:47.449114 5998 patch_prober.go:29] interesting pod/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 02:03:47.454469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:47.449189 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" podUID="0ad87fa4-78c9-4e60-8706-a78dbd1002b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 02:03:47.455163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:47.455132 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:47.778168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:47.777937 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:48.372377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:48.372341 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:48.451113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:48.447473 5998 patch_prober.go:29] interesting pod/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 02:03:48.451113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:48.447534 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" podUID="0ad87fa4-78c9-4e60-8706-a78dbd1002b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 02:03:49.193885 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:49.193849 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:49.447793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:49.447705 5998 patch_prober.go:29] interesting pod/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 02:03:49.447793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:49.447762 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" podUID="0ad87fa4-78c9-4e60-8706-a78dbd1002b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 02:03:49.582891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:49.582851 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:50.052080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.052044 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s"] Oct 30 02:03:50.056302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.056270 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.092142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.090480 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-3ad281-openshift-default-dockercfg-s4h6l\"" Oct 30 02:03:50.093682 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.093651 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s"] Oct 30 02:03:50.161955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.161915 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prbt9\" (UniqueName: \"kubernetes.io/projected/b24b4bd4-4eb0-412c-a54c-250a72795d94-kube-api-access-prbt9\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.172476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.172440 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-token\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.172672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.172655 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-envoy\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.172784 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.172770 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-podinfo\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.172959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.172943 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-credential-socket\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.173074 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.173060 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-workload-socket\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.173329 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.173309 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-workload-certs\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.173571 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.173554 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-data\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.173764 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.173745 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24b4bd4-4eb0-412c-a54c-250a72795d94-istiod-ca-cert\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.253121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.245639 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:50.278115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.276920 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-token\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.278115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.276976 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-envoy\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.278115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.277007 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-podinfo\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.278115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.277079 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-credential-socket\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.278115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.277134 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-workload-socket\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.278115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.277201 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-workload-certs\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.278115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.277258 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-data\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.278115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.277294 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24b4bd4-4eb0-412c-a54c-250a72795d94-istiod-ca-cert\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.278115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.277416 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prbt9\" (UniqueName: \"kubernetes.io/projected/b24b4bd4-4eb0-412c-a54c-250a72795d94-kube-api-access-prbt9\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.285788 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.285756 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-workload-socket\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.296160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.295156 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-data\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.471128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.295831 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24b4bd4-4eb0-412c-a54c-250a72795d94-istiod-ca-cert\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.471413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.296057 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-credential-socket\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.471522 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.328961 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-podinfo\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.471609 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.422546 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prbt9\" (UniqueName: \"kubernetes.io/projected/b24b4bd4-4eb0-412c-a54c-250a72795d94-kube-api-access-prbt9\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.471696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.427928 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:50.471869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.448754 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-token\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.471978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.460701 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-workload-certs\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.476779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.475377 5998 patch_prober.go:29] interesting pod/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 02:03:50.476779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.475435 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" podUID="0ad87fa4-78c9-4e60-8706-a78dbd1002b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 02:03:50.504142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.503450 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-envoy\") pod \"e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.672916 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.672877 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:50.931114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.930364 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:03:50.995114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:50.994904 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:03:51.065021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:51.064929 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:51.256461 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:51.256429 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s"] Oct 30 02:03:51.261890 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:03:51.261858 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24b4bd4_4eb0_412c_a54c_250a72795d94.slice/crio-9b16f0c9956d6c1fd187b6dfbdc2a94d01b0d95d503b3217e7d44fab04101f4a WatchSource:0}: Error finding container 9b16f0c9956d6c1fd187b6dfbdc2a94d01b0d95d503b3217e7d44fab04101f4a: Status 404 returned error can't find the container with id 9b16f0c9956d6c1fd187b6dfbdc2a94d01b0d95d503b3217e7d44fab04101f4a Oct 30 02:03:51.268544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:51.268511 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:03:51.268641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:51.268579 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:03:51.268641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:51.268609 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:03:51.452571 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:51.452540 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:03:51.583160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:51.583123 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" event={"ID":"b24b4bd4-4eb0-412c-a54c-250a72795d94","Type":"ContainerStarted","Data":"c5ff79cac9535ecffdca44eb0a19caf2fc937f93bdb78b51ea57a2b28f69cac0"} Oct 30 02:03:51.583366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:51.583189 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" event={"ID":"b24b4bd4-4eb0-412c-a54c-250a72795d94","Type":"ContainerStarted","Data":"9b16f0c9956d6c1fd187b6dfbdc2a94d01b0d95d503b3217e7d44fab04101f4a"} Oct 30 02:03:51.584929 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:51.584861 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:03:51.636123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:51.636040 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" podStartSLOduration=2.6360193929999998 podStartE2EDuration="2.636019393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:03:51.631481556 +0000 UTC m=+2061.210272640" watchObservedRunningTime="2026-10-30 02:03:51.636019393 +0000 UTC m=+2061.214810476" Oct 30 02:03:51.673770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:51.673722 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:52.680049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:52.680009 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:52.980964 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:52.980871 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:53.195350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:53.195219 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:53.596398 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:53.596370 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:53.597334 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:53.597317 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:03:54.053354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:54.053315 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:54.178524 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:54.178419 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:54.811436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:54.811401 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:55.222314 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:03:55.222277 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:03:55.780156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:55.780124 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:55.893234 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:55.893203 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:57.021119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:57.021048 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:57.158058 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:57.158004 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:58.732799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:58.732762 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:59.352469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:59.352437 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:59.875789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:59.875759 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:03:59.988421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:03:59.988373 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:00.394575 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:00.394539 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:00.828927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:00.828887 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:02.154827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:02.154793 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:04.293437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:04.293344 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:04.587558 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:04.587464 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:06.149434 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:06.149390 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:06.645140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:06.645061 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:07.878045 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:07.878001 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:09.472170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:09.472116 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:10.302701 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:10.302660 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:11.162467 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:11.162424 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:12.261533 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:12.261424 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:12.760349 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:12.760282 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:14.172893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:14.172839 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:14.280676 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:14.280640 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:14.568626 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:14.568544 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:14.921119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:14.921029 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:04:15.356276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:15.356184 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:17.033116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:17.033056 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:17.179620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:17.179584 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:17.373950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:17.373860 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:19.164179 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:19.164143 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:19.901307 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:19.901275 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:20.036223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:20.036193 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:20.630130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:20.628721 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:21.552746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:21.552658 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:23.001954 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:23.001917 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:23.132353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:23.132318 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:23.752032 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:23.751998 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:24.452422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:24.452384 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:25.232048 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:04:25.232010 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:04:25.759325 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:25.759285 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:27.456319 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:27.456282 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:27.569140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:27.569085 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:28.567320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:28.567267 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:28.674521 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:28.674485 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:28.831653 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:28.831542 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:29.261116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:29.257586 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:29.915742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:29.915702 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp"] Oct 30 02:04:29.919594 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:29.919574 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp" Oct 30 02:04:29.934977 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:29.934955 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:04:29.935147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:29.935021 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:04:29.971716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:29.971681 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7jwz\" (UniqueName: \"kubernetes.io/projected/8c8baaad-047f-4855-865c-ba855adb52c9-kube-api-access-m7jwz\") pod \"maas-api-revoke-sub-83656716-h2nwp\" (UID: \"8c8baaad-047f-4855-865c-ba855adb52c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp" Oct 30 02:04:29.971911 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:29.971804 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8baaad-047f-4855-865c-ba855adb52c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83656716-h2nwp\" (UID: \"8c8baaad-047f-4855-865c-ba855adb52c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp" Oct 30 02:04:29.971911 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:29.971858 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8baaad-047f-4855-865c-ba855adb52c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83656716-h2nwp\" (UID: \"8c8baaad-047f-4855-865c-ba855adb52c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp" Oct 30 02:04:29.978318 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:29.978286 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp"] Oct 30 02:04:30.013021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:30.012994 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:04:30.017022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:30.016990 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:04:30.073053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:30.073017 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7jwz\" (UniqueName: \"kubernetes.io/projected/8c8baaad-047f-4855-865c-ba855adb52c9-kube-api-access-m7jwz\") pod \"maas-api-revoke-sub-83656716-h2nwp\" (UID: \"8c8baaad-047f-4855-865c-ba855adb52c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp" Oct 30 02:04:30.073247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:30.073136 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8baaad-047f-4855-865c-ba855adb52c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83656716-h2nwp\" (UID: \"8c8baaad-047f-4855-865c-ba855adb52c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp" Oct 30 02:04:30.073247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:30.073188 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8baaad-047f-4855-865c-ba855adb52c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83656716-h2nwp\" (UID: \"8c8baaad-047f-4855-865c-ba855adb52c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp" Oct 30 02:04:30.073821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:30.073793 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8baaad-047f-4855-865c-ba855adb52c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83656716-h2nwp\" (UID: \"8c8baaad-047f-4855-865c-ba855adb52c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp" Oct 30 02:04:30.075567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:30.075541 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8baaad-047f-4855-865c-ba855adb52c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83656716-h2nwp\" (UID: \"8c8baaad-047f-4855-865c-ba855adb52c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp" Oct 30 02:04:30.097064 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:30.097038 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7jwz\" (UniqueName: \"kubernetes.io/projected/8c8baaad-047f-4855-865c-ba855adb52c9-kube-api-access-m7jwz\") pod \"maas-api-revoke-sub-83656716-h2nwp\" (UID: \"8c8baaad-047f-4855-865c-ba855adb52c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp" Oct 30 02:04:30.239819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:30.239480 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp" Oct 30 02:04:30.638546 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:30.638509 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp"] Oct 30 02:04:30.640885 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:04:30.640851 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8baaad_047f_4855_865c_ba855adb52c9.slice/crio-d603cd1f7b9805833b220f506d803e1606b093dcb36c9e78a024dcbf7a9b1cc8 WatchSource:0}: Error finding container d603cd1f7b9805833b220f506d803e1606b093dcb36c9e78a024dcbf7a9b1cc8: Status 404 returned error can't find the container with id d603cd1f7b9805833b220f506d803e1606b093dcb36c9e78a024dcbf7a9b1cc8 Oct 30 02:04:30.909503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:30.909469 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp" event={"ID":"8c8baaad-047f-4855-865c-ba855adb52c9","Type":"ContainerStarted","Data":"d603cd1f7b9805833b220f506d803e1606b093dcb36c9e78a024dcbf7a9b1cc8"} Oct 30 02:04:31.927981 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:31.927937 5998 generic.go:356] "Generic (PLEG): container finished" podID="8c8baaad-047f-4855-865c-ba855adb52c9" containerID="b469979ed5190efdb5ad7f47aa71f54eeafaf4cd78d2513811348cfeff5aa0e6" exitCode=0 Oct 30 02:04:31.928530 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:31.928118 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp" event={"ID":"8c8baaad-047f-4855-865c-ba855adb52c9","Type":"ContainerDied","Data":"b469979ed5190efdb5ad7f47aa71f54eeafaf4cd78d2513811348cfeff5aa0e6"} Oct 30 02:04:32.005743 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:32.005543 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:33.152630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:33.152596 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp" Oct 30 02:04:33.262869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:33.262781 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8baaad-047f-4855-865c-ba855adb52c9-kube-api-access-m7jwz\" (UniqueName: \"kubernetes.io/projected/8c8baaad-047f-4855-865c-ba855adb52c9-kube-api-access-m7jwz\") pod \"8c8baaad-047f-4855-865c-ba855adb52c9\" (UID: \"8c8baaad-047f-4855-865c-ba855adb52c9\") " Oct 30 02:04:33.262869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:33.262841 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8baaad-047f-4855-865c-ba855adb52c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8baaad-047f-4855-865c-ba855adb52c9-maas-api-openshift-ca\") pod \"8c8baaad-047f-4855-865c-ba855adb52c9\" (UID: \"8c8baaad-047f-4855-865c-ba855adb52c9\") " Oct 30 02:04:33.263153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:33.262900 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c8baaad-047f-4855-865c-ba855adb52c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8baaad-047f-4855-865c-ba855adb52c9-maas-api-xks-ca\") pod \"8c8baaad-047f-4855-865c-ba855adb52c9\" (UID: \"8c8baaad-047f-4855-865c-ba855adb52c9\") " Oct 30 02:04:33.263234 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:33.263174 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8baaad-047f-4855-865c-ba855adb52c9-maas-api-openshift-ca" pod "8c8baaad-047f-4855-865c-ba855adb52c9" (UID: "8c8baaad-047f-4855-865c-ba855adb52c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:33.263321 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:33.263303 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8baaad-047f-4855-865c-ba855adb52c9-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:33.264958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:33.264933 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c8baaad-047f-4855-865c-ba855adb52c9-maas-api-xks-ca" pod "8c8baaad-047f-4855-865c-ba855adb52c9" (UID: "8c8baaad-047f-4855-865c-ba855adb52c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:04:33.265075 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:33.264933 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8baaad-047f-4855-865c-ba855adb52c9-kube-api-access-m7jwz" pod "8c8baaad-047f-4855-865c-ba855adb52c9" (UID: "8c8baaad-047f-4855-865c-ba855adb52c9"). InnerVolumeSpecName "kube-api-access-m7jwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:33.364214 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:33.364178 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7jwz\" (UniqueName: \"kubernetes.io/projected/8c8baaad-047f-4855-865c-ba855adb52c9-kube-api-access-m7jwz\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:33.364214 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:33.364210 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8baaad-047f-4855-865c-ba855adb52c9-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:33.666278 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:33.666243 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:33.946767 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:33.946688 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp" Oct 30 02:04:33.947023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:33.946687 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp" event={"ID":"8c8baaad-047f-4855-865c-ba855adb52c9","Type":"ContainerDied","Data":"d603cd1f7b9805833b220f506d803e1606b093dcb36c9e78a024dcbf7a9b1cc8"} Oct 30 02:04:33.947232 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:33.947216 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d603cd1f7b9805833b220f506d803e1606b093dcb36c9e78a024dcbf7a9b1cc8" Oct 30 02:04:35.264700 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:35.264660 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:35.975696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:35.975667 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:36.621160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:36.621121 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:37.410624 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:37.410595 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:39.116644 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:39.116606 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:39.637311 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:39.637263 5998 patch_prober.go:29] interesting pod/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:04:39.637498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:39.637340 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" podUID="b24b4bd4-4eb0-412c-a54c-250a72795d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:04:41.001788 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:41.001754 5998 generic.go:356] "Generic (PLEG): container finished" podID="b24b4bd4-4eb0-412c-a54c-250a72795d94" containerID="c5ff79cac9535ecffdca44eb0a19caf2fc937f93bdb78b51ea57a2b28f69cac0" exitCode=0 Oct 30 02:04:41.002302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:41.001833 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" event={"ID":"b24b4bd4-4eb0-412c-a54c-250a72795d94","Type":"ContainerDied","Data":"c5ff79cac9535ecffdca44eb0a19caf2fc937f93bdb78b51ea57a2b28f69cac0"} Oct 30 02:04:41.002302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:41.002249 5998 scope.go:122] "RemoveContainer" containerID="c5ff79cac9535ecffdca44eb0a19caf2fc937f93bdb78b51ea57a2b28f69cac0" Oct 30 02:04:41.003813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:41.003782 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:04:41.003930 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:41.003853 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:04:41.003930 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:41.003886 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:04:41.192586 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:41.192558 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:41.964274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:41.964192 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:42.082130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.081477 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" event={"ID":"b24b4bd4-4eb0-412c-a54c-250a72795d94","Type":"ContainerStarted","Data":"065632ed5247eef5acd54bcd4df4e5ad60b7fcccb360d758a5adf03310af8ff8"} Oct 30 02:04:42.673190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.673151 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj"] Oct 30 02:04:42.674805 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.674771 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8c8baaad-047f-4855-865c-ba855adb52c9" containerName="revoke-keys" Oct 30 02:04:42.674976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.674962 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8baaad-047f-4855-865c-ba855adb52c9" containerName="revoke-keys" Oct 30 02:04:42.675421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.675401 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8baaad-047f-4855-865c-ba855adb52c9" containerName="revoke-keys" Oct 30 02:04:42.684042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.680078 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj" Oct 30 02:04:42.735748 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.726493 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d397d80-sxgkj\" (UID: \"cd1b3bc6-a41a-4244-a7eb-3389898e88d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj" Oct 30 02:04:42.735748 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.726620 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgg2d\" (UniqueName: \"kubernetes.io/projected/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-kube-api-access-mgg2d\") pod \"maas-api-revoke-sub-6d397d80-sxgkj\" (UID: \"cd1b3bc6-a41a-4244-a7eb-3389898e88d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj" Oct 30 02:04:42.735748 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.726684 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d397d80-sxgkj\" (UID: \"cd1b3bc6-a41a-4244-a7eb-3389898e88d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj" Oct 30 02:04:42.744871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.737229 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:04:42.758207 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.758051 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:04:42.758207 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.758205 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:04:42.758497 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.758407 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:04:42.758497 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.758460 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:04:42.758633 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.758502 5998 patch_prober.go:29] interesting pod/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 02:04:42.758633 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.758594 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" podUID="b24b4bd4-4eb0-412c-a54c-250a72795d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 02:04:42.760874 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.760851 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj"] Oct 30 02:04:42.828221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.828172 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgg2d\" (UniqueName: \"kubernetes.io/projected/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-kube-api-access-mgg2d\") pod \"maas-api-revoke-sub-6d397d80-sxgkj\" (UID: \"cd1b3bc6-a41a-4244-a7eb-3389898e88d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj" Oct 30 02:04:42.828439 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.828274 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d397d80-sxgkj\" (UID: \"cd1b3bc6-a41a-4244-a7eb-3389898e88d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj" Oct 30 02:04:42.828439 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.828384 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d397d80-sxgkj\" (UID: \"cd1b3bc6-a41a-4244-a7eb-3389898e88d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj" Oct 30 02:04:42.830085 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.830055 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d397d80-sxgkj\" (UID: \"cd1b3bc6-a41a-4244-a7eb-3389898e88d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj" Oct 30 02:04:42.831198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.831166 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d397d80-sxgkj\" (UID: \"cd1b3bc6-a41a-4244-a7eb-3389898e88d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj" Oct 30 02:04:42.851060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:42.851032 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgg2d\" (UniqueName: \"kubernetes.io/projected/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-kube-api-access-mgg2d\") pod \"maas-api-revoke-sub-6d397d80-sxgkj\" (UID: \"cd1b3bc6-a41a-4244-a7eb-3389898e88d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj" Oct 30 02:04:43.029943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:43.029849 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n"] Oct 30 02:04:43.030157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:43.030120 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" podUID="321d663c-9621-4376-9ef9-f8aeeafa3f20" containerName="istio-proxy" containerID="cri-o://d1567a028e016acdee7e90f00c17b498b730c1e4ebcda5f7269d2ad1cb2bd597" gracePeriod=30 Oct 30 02:04:43.072743 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:43.072707 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj" Oct 30 02:04:43.333524 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:43.333439 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:43.452072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:43.452042 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj"] Oct 30 02:04:43.456175 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:04:43.456146 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd1b3bc6_a41a_4244_a7eb_3389898e88d2.slice/crio-75e17a1e2ca5e8cdc61a337c74f05a855c432211b3f5967cd8aaf98915fd94ce WatchSource:0}: Error finding container 75e17a1e2ca5e8cdc61a337c74f05a855c432211b3f5967cd8aaf98915fd94ce: Status 404 returned error can't find the container with id 75e17a1e2ca5e8cdc61a337c74f05a855c432211b3f5967cd8aaf98915fd94ce Oct 30 02:04:43.674222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:43.674185 5998 patch_prober.go:29] interesting pod/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 02:04:43.674419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:43.674255 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" podUID="b24b4bd4-4eb0-412c-a54c-250a72795d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 02:04:43.869417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:43.869378 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:44.100009 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:44.099966 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj" event={"ID":"cd1b3bc6-a41a-4244-a7eb-3389898e88d2","Type":"ContainerStarted","Data":"d28cfdda14a0cf8bb4690873dbe60b4a4f2d397632a32f04449a893042f690d5"} Oct 30 02:04:44.100223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:44.100032 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj" event={"ID":"cd1b3bc6-a41a-4244-a7eb-3389898e88d2","Type":"ContainerStarted","Data":"75e17a1e2ca5e8cdc61a337c74f05a855c432211b3f5967cd8aaf98915fd94ce"} Oct 30 02:04:44.144215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:44.144156 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj" podStartSLOduration=2.14414054 podStartE2EDuration="2.14414054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:04:44.140682638 +0000 UTC m=+2113.719473720" watchObservedRunningTime="2026-10-30 02:04:44.14414054 +0000 UTC m=+2113.722931620" Oct 30 02:04:44.673431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:44.673396 5998 patch_prober.go:29] interesting pod/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 02:04:44.673830 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:44.673448 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" podUID="b24b4bd4-4eb0-412c-a54c-250a72795d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 02:04:45.010809 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:45.010752 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:45.150125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:45.147763 5998 generic.go:356] "Generic (PLEG): container finished" podID="cd1b3bc6-a41a-4244-a7eb-3389898e88d2" containerID="d28cfdda14a0cf8bb4690873dbe60b4a4f2d397632a32f04449a893042f690d5" exitCode=0 Oct 30 02:04:45.150125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:45.147922 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj" event={"ID":"cd1b3bc6-a41a-4244-a7eb-3389898e88d2","Type":"ContainerDied","Data":"d28cfdda14a0cf8bb4690873dbe60b4a4f2d397632a32f04449a893042f690d5"} Oct 30 02:04:45.181893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:45.181858 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:45.304192 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:45.300389 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:45.673858 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:45.673822 5998 patch_prober.go:29] interesting pod/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 02:04:45.674296 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:45.673885 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" podUID="b24b4bd4-4eb0-412c-a54c-250a72795d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 02:04:45.832712 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:45.832675 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:46.287862 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.287811 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n"] Oct 30 02:04:46.288706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.288663 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" podUID="41302545-fcf3-4f12-80b1-14d1992bedf5" containerName="istio-proxy" containerID="cri-o://a6b7c49602c455bafdd28b55ddc97d0e9e274726f3d236f68a424b26cff1f24a" gracePeriod=30 Oct 30 02:04:46.346200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.346167 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:46.589978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.589940 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj" Oct 30 02:04:46.670985 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.668508 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-kube-api-access-mgg2d\" (UniqueName: \"kubernetes.io/projected/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-kube-api-access-mgg2d\") pod \"cd1b3bc6-a41a-4244-a7eb-3389898e88d2\" (UID: \"cd1b3bc6-a41a-4244-a7eb-3389898e88d2\") " Oct 30 02:04:46.670985 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.668557 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-maas-api-openshift-ca\") pod \"cd1b3bc6-a41a-4244-a7eb-3389898e88d2\" (UID: \"cd1b3bc6-a41a-4244-a7eb-3389898e88d2\") " Oct 30 02:04:46.670985 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.668963 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-maas-api-xks-ca\") pod \"cd1b3bc6-a41a-4244-a7eb-3389898e88d2\" (UID: \"cd1b3bc6-a41a-4244-a7eb-3389898e88d2\") " Oct 30 02:04:46.670985 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.669017 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-maas-api-openshift-ca" pod "cd1b3bc6-a41a-4244-a7eb-3389898e88d2" (UID: "cd1b3bc6-a41a-4244-a7eb-3389898e88d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:46.670985 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.669832 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:46.673673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.673644 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-maas-api-xks-ca" pod "cd1b3bc6-a41a-4244-a7eb-3389898e88d2" (UID: "cd1b3bc6-a41a-4244-a7eb-3389898e88d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:04:46.674119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.674077 5998 patch_prober.go:29] interesting pod/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 02:04:46.674510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.674156 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" podUID="b24b4bd4-4eb0-412c-a54c-250a72795d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 02:04:46.674777 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.674746 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-kube-api-access-mgg2d" pod "cd1b3bc6-a41a-4244-a7eb-3389898e88d2" (UID: "cd1b3bc6-a41a-4244-a7eb-3389898e88d2"). InnerVolumeSpecName "kube-api-access-mgg2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:46.771397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.771358 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgg2d\" (UniqueName: \"kubernetes.io/projected/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-kube-api-access-mgg2d\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:46.771397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.771390 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd1b3bc6-a41a-4244-a7eb-3389898e88d2-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:46.845436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.845351 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc"] Oct 30 02:04:46.846002 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.845980 5998 cpu_manager.go:414] "containerMap: removing container" podUID="cd1b3bc6-a41a-4244-a7eb-3389898e88d2" containerName="revoke-keys" Oct 30 02:04:46.846144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.846003 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd1b3bc6-a41a-4244-a7eb-3389898e88d2" containerName="revoke-keys" Oct 30 02:04:46.846224 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.846178 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd1b3bc6-a41a-4244-a7eb-3389898e88d2" containerName="revoke-keys" Oct 30 02:04:46.849681 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.849657 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc" Oct 30 02:04:46.871859 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.869801 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc"] Oct 30 02:04:46.973881 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.973839 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb00c574-0b03-48f4-a555-117ed93a9159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e66dcbaa-7b4fc\" (UID: \"eb00c574-0b03-48f4-a555-117ed93a9159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc" Oct 30 02:04:46.974072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.973898 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb00c574-0b03-48f4-a555-117ed93a9159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e66dcbaa-7b4fc\" (UID: \"eb00c574-0b03-48f4-a555-117ed93a9159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc" Oct 30 02:04:46.974072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:46.974021 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwfx\" (UniqueName: \"kubernetes.io/projected/eb00c574-0b03-48f4-a555-117ed93a9159-kube-api-access-mbwfx\") pod \"maas-api-revoke-sub-e66dcbaa-7b4fc\" (UID: \"eb00c574-0b03-48f4-a555-117ed93a9159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc" Oct 30 02:04:47.026179 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:47.026129 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:47.075476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:47.075439 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb00c574-0b03-48f4-a555-117ed93a9159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e66dcbaa-7b4fc\" (UID: \"eb00c574-0b03-48f4-a555-117ed93a9159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc" Oct 30 02:04:47.075663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:47.075489 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb00c574-0b03-48f4-a555-117ed93a9159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e66dcbaa-7b4fc\" (UID: \"eb00c574-0b03-48f4-a555-117ed93a9159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc" Oct 30 02:04:47.075663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:47.075537 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwfx\" (UniqueName: \"kubernetes.io/projected/eb00c574-0b03-48f4-a555-117ed93a9159-kube-api-access-mbwfx\") pod \"maas-api-revoke-sub-e66dcbaa-7b4fc\" (UID: \"eb00c574-0b03-48f4-a555-117ed93a9159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc" Oct 30 02:04:47.076246 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:47.076219 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb00c574-0b03-48f4-a555-117ed93a9159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e66dcbaa-7b4fc\" (UID: \"eb00c574-0b03-48f4-a555-117ed93a9159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc" Oct 30 02:04:47.077929 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:47.077891 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb00c574-0b03-48f4-a555-117ed93a9159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e66dcbaa-7b4fc\" (UID: \"eb00c574-0b03-48f4-a555-117ed93a9159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc" Oct 30 02:04:47.099561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:47.099488 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwfx\" (UniqueName: \"kubernetes.io/projected/eb00c574-0b03-48f4-a555-117ed93a9159-kube-api-access-mbwfx\") pod \"maas-api-revoke-sub-e66dcbaa-7b4fc\" (UID: \"eb00c574-0b03-48f4-a555-117ed93a9159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc" Oct 30 02:04:47.163314 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:47.163281 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj" event={"ID":"cd1b3bc6-a41a-4244-a7eb-3389898e88d2","Type":"ContainerDied","Data":"75e17a1e2ca5e8cdc61a337c74f05a855c432211b3f5967cd8aaf98915fd94ce"} Oct 30 02:04:47.163314 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:47.163309 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj" Oct 30 02:04:47.163545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:47.163331 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75e17a1e2ca5e8cdc61a337c74f05a855c432211b3f5967cd8aaf98915fd94ce" Oct 30 02:04:47.165449 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:47.165430 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc" Oct 30 02:04:47.183922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:47.183894 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:47.518229 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:47.517850 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc"] Oct 30 02:04:47.522058 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:04:47.521180 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb00c574_0b03_48f4_a555_117ed93a9159.slice/crio-805082f26fd0243ebe15e010ab6508bffd6adf57431be54d434e3434cf6f447f WatchSource:0}: Error finding container 805082f26fd0243ebe15e010ab6508bffd6adf57431be54d434e3434cf6f447f: Status 404 returned error can't find the container with id 805082f26fd0243ebe15e010ab6508bffd6adf57431be54d434e3434cf6f447f Oct 30 02:04:47.803538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:47.803448 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:04:47.803998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:47.803871 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:04:47.804938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:47.804909 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:04:48.173576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.173518 5998 generic.go:356] "Generic (PLEG): container finished" podID="321d663c-9621-4376-9ef9-f8aeeafa3f20" containerID="d1567a028e016acdee7e90f00c17b498b730c1e4ebcda5f7269d2ad1cb2bd597" exitCode=0 Oct 30 02:04:48.174024 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.173802 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" event={"ID":"321d663c-9621-4376-9ef9-f8aeeafa3f20","Type":"ContainerDied","Data":"d1567a028e016acdee7e90f00c17b498b730c1e4ebcda5f7269d2ad1cb2bd597"} Oct 30 02:04:48.177082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.177052 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc" event={"ID":"eb00c574-0b03-48f4-a555-117ed93a9159","Type":"ContainerStarted","Data":"de625751a900d92dc2b14e310a10d5d628f011dcf302176155d4a2b3b6333c18"} Oct 30 02:04:48.177315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.177116 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc" event={"ID":"eb00c574-0b03-48f4-a555-117ed93a9159","Type":"ContainerStarted","Data":"805082f26fd0243ebe15e010ab6508bffd6adf57431be54d434e3434cf6f447f"} Oct 30 02:04:48.220677 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.217046 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc" podStartSLOduration=2.2170269129999998 podStartE2EDuration="2.217026913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:04:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:04:48.214859548 +0000 UTC m=+2117.793650629" watchObservedRunningTime="2026-10-30 02:04:48.217026913 +0000 UTC m=+2117.795817994" Oct 30 02:04:48.250132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.246505 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:48.357394 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.357358 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 02:04:48.381081 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.381043 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:48.500217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.500174 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-podinfo\") pod \"321d663c-9621-4376-9ef9-f8aeeafa3f20\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " Oct 30 02:04:48.500436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.500244 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321d663c-9621-4376-9ef9-f8aeeafa3f20-kube-api-access-dpmxv\" (UniqueName: \"kubernetes.io/projected/321d663c-9621-4376-9ef9-f8aeeafa3f20-kube-api-access-dpmxv\") pod \"321d663c-9621-4376-9ef9-f8aeeafa3f20\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " Oct 30 02:04:48.500436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.500294 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-envoy\") pod \"321d663c-9621-4376-9ef9-f8aeeafa3f20\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " Oct 30 02:04:48.500436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.500341 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-workload-socket\") pod \"321d663c-9621-4376-9ef9-f8aeeafa3f20\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " Oct 30 02:04:48.500436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.500398 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321d663c-9621-4376-9ef9-f8aeeafa3f20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321d663c-9621-4376-9ef9-f8aeeafa3f20-istiod-ca-cert\") pod \"321d663c-9621-4376-9ef9-f8aeeafa3f20\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " Oct 30 02:04:48.500660 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.500536 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-workload-certs\") pod \"321d663c-9621-4376-9ef9-f8aeeafa3f20\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " Oct 30 02:04:48.500660 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.500629 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-token\" (UniqueName: \"kubernetes.io/projected/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-token\") pod \"321d663c-9621-4376-9ef9-f8aeeafa3f20\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " Oct 30 02:04:48.500781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.500663 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-data\") pod \"321d663c-9621-4376-9ef9-f8aeeafa3f20\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " Oct 30 02:04:48.500781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.500694 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-credential-socket\") pod \"321d663c-9621-4376-9ef9-f8aeeafa3f20\" (UID: \"321d663c-9621-4376-9ef9-f8aeeafa3f20\") " Oct 30 02:04:48.500781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.500698 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-workload-socket" pod "321d663c-9621-4376-9ef9-f8aeeafa3f20" (UID: "321d663c-9621-4376-9ef9-f8aeeafa3f20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:48.500943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.500874 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-workload-certs" pod "321d663c-9621-4376-9ef9-f8aeeafa3f20" (UID: "321d663c-9621-4376-9ef9-f8aeeafa3f20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:48.501152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.501127 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-credential-socket" pod "321d663c-9621-4376-9ef9-f8aeeafa3f20" (UID: "321d663c-9621-4376-9ef9-f8aeeafa3f20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:48.501359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.501286 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-data" pod "321d663c-9621-4376-9ef9-f8aeeafa3f20" (UID: "321d663c-9621-4376-9ef9-f8aeeafa3f20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:48.501518 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.501418 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321d663c-9621-4376-9ef9-f8aeeafa3f20-istiod-ca-cert" pod "321d663c-9621-4376-9ef9-f8aeeafa3f20" (UID: "321d663c-9621-4376-9ef9-f8aeeafa3f20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:48.501518 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.501416 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:48.501682 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.501574 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:48.502910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.502884 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-podinfo" pod "321d663c-9621-4376-9ef9-f8aeeafa3f20" (UID: "321d663c-9621-4376-9ef9-f8aeeafa3f20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:04:48.503060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.503035 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321d663c-9621-4376-9ef9-f8aeeafa3f20-kube-api-access-dpmxv" pod "321d663c-9621-4376-9ef9-f8aeeafa3f20" (UID: "321d663c-9621-4376-9ef9-f8aeeafa3f20"). InnerVolumeSpecName "kube-api-access-dpmxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:48.503411 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.503367 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-token" pod "321d663c-9621-4376-9ef9-f8aeeafa3f20" (UID: "321d663c-9621-4376-9ef9-f8aeeafa3f20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:48.503411 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.503367 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-envoy" pod "321d663c-9621-4376-9ef9-f8aeeafa3f20" (UID: "321d663c-9621-4376-9ef9-f8aeeafa3f20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:48.625174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.624137 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:48.625174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.624174 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321d663c-9621-4376-9ef9-f8aeeafa3f20-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:48.625174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.624188 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:48.625174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.624201 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:48.625174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.624214 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321d663c-9621-4376-9ef9-f8aeeafa3f20-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:48.625174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.624228 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321d663c-9621-4376-9ef9-f8aeeafa3f20-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:48.625174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:48.624242 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpmxv\" (UniqueName: \"kubernetes.io/projected/321d663c-9621-4376-9ef9-f8aeeafa3f20-kube-api-access-dpmxv\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:49.185233 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:49.185201 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" Oct 30 02:04:49.185745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:49.185203 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n" event={"ID":"321d663c-9621-4376-9ef9-f8aeeafa3f20","Type":"ContainerDied","Data":"01663ffeeaadb2063a5a1dbf81aa5940dbe1ce3e20badeac95e1b34291bb1db0"} Oct 30 02:04:49.185745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:49.185396 5998 scope.go:122] "RemoveContainer" containerID="d1567a028e016acdee7e90f00c17b498b730c1e4ebcda5f7269d2ad1cb2bd597" Oct 30 02:04:49.188944 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:49.188913 5998 generic.go:356] "Generic (PLEG): container finished" podID="eb00c574-0b03-48f4-a555-117ed93a9159" containerID="de625751a900d92dc2b14e310a10d5d628f011dcf302176155d4a2b3b6333c18" exitCode=0 Oct 30 02:04:49.189067 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:49.189005 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc" event={"ID":"eb00c574-0b03-48f4-a555-117ed93a9159","Type":"ContainerDied","Data":"de625751a900d92dc2b14e310a10d5d628f011dcf302176155d4a2b3b6333c18"} Oct 30 02:04:49.247211 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:49.247182 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n"] Oct 30 02:04:49.251471 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:49.251441 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-zcj7n"] Oct 30 02:04:49.279306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:49.279278 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:50.011312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:50.011271 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:50.382805 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:50.382777 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc" Oct 30 02:04:50.539442 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:50.539406 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321d663c-9621-4376-9ef9-f8aeeafa3f20" path="/var/lib/kubelet/pods/321d663c-9621-4376-9ef9-f8aeeafa3f20/volumes" Oct 30 02:04:50.551333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:50.551291 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb00c574-0b03-48f4-a555-117ed93a9159-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb00c574-0b03-48f4-a555-117ed93a9159-maas-api-openshift-ca\") pod \"eb00c574-0b03-48f4-a555-117ed93a9159\" (UID: \"eb00c574-0b03-48f4-a555-117ed93a9159\") " Oct 30 02:04:50.551493 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:50.551455 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb00c574-0b03-48f4-a555-117ed93a9159-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb00c574-0b03-48f4-a555-117ed93a9159-maas-api-xks-ca\") pod \"eb00c574-0b03-48f4-a555-117ed93a9159\" (UID: \"eb00c574-0b03-48f4-a555-117ed93a9159\") " Oct 30 02:04:50.551493 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:50.551488 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb00c574-0b03-48f4-a555-117ed93a9159-kube-api-access-mbwfx\" (UniqueName: \"kubernetes.io/projected/eb00c574-0b03-48f4-a555-117ed93a9159-kube-api-access-mbwfx\") pod \"eb00c574-0b03-48f4-a555-117ed93a9159\" (UID: \"eb00c574-0b03-48f4-a555-117ed93a9159\") " Oct 30 02:04:50.551675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:50.551649 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb00c574-0b03-48f4-a555-117ed93a9159-maas-api-openshift-ca" pod "eb00c574-0b03-48f4-a555-117ed93a9159" (UID: "eb00c574-0b03-48f4-a555-117ed93a9159"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:50.551905 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:50.551887 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb00c574-0b03-48f4-a555-117ed93a9159-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:50.553648 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:50.553619 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb00c574-0b03-48f4-a555-117ed93a9159-maas-api-xks-ca" pod "eb00c574-0b03-48f4-a555-117ed93a9159" (UID: "eb00c574-0b03-48f4-a555-117ed93a9159"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:04:50.553648 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:50.553620 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb00c574-0b03-48f4-a555-117ed93a9159-kube-api-access-mbwfx" pod "eb00c574-0b03-48f4-a555-117ed93a9159" (UID: "eb00c574-0b03-48f4-a555-117ed93a9159"). InnerVolumeSpecName "kube-api-access-mbwfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:50.653816 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:50.653729 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb00c574-0b03-48f4-a555-117ed93a9159-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:50.653816 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:50.653768 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbwfx\" (UniqueName: \"kubernetes.io/projected/eb00c574-0b03-48f4-a555-117ed93a9159-kube-api-access-mbwfx\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:50.976982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:50.976946 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:51.082978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.082944 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:51.207215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.207182 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc" Oct 30 02:04:51.207215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.207190 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc" event={"ID":"eb00c574-0b03-48f4-a555-117ed93a9159","Type":"ContainerDied","Data":"805082f26fd0243ebe15e010ab6508bffd6adf57431be54d434e3434cf6f447f"} Oct 30 02:04:51.207497 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.207251 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="805082f26fd0243ebe15e010ab6508bffd6adf57431be54d434e3434cf6f447f" Oct 30 02:04:51.582418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.582381 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:04:51.664864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.664828 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-envoy\") pod \"41302545-fcf3-4f12-80b1-14d1992bedf5\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " Oct 30 02:04:51.665065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.664891 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-credential-socket\") pod \"41302545-fcf3-4f12-80b1-14d1992bedf5\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " Oct 30 02:04:51.665065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.664992 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-podinfo\") pod \"41302545-fcf3-4f12-80b1-14d1992bedf5\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " Oct 30 02:04:51.665065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.665056 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41302545-fcf3-4f12-80b1-14d1992bedf5-kube-api-access-4hpxw\" (UniqueName: \"kubernetes.io/projected/41302545-fcf3-4f12-80b1-14d1992bedf5-kube-api-access-4hpxw\") pod \"41302545-fcf3-4f12-80b1-14d1992bedf5\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " Oct 30 02:04:51.665269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.665146 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-data\") pod \"41302545-fcf3-4f12-80b1-14d1992bedf5\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " Oct 30 02:04:51.665269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.665216 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-token\" (UniqueName: \"kubernetes.io/projected/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-token\") pod \"41302545-fcf3-4f12-80b1-14d1992bedf5\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " Oct 30 02:04:51.665391 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.665266 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41302545-fcf3-4f12-80b1-14d1992bedf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41302545-fcf3-4f12-80b1-14d1992bedf5-istiod-ca-cert\") pod \"41302545-fcf3-4f12-80b1-14d1992bedf5\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " Oct 30 02:04:51.665391 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.665273 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-credential-socket" pod "41302545-fcf3-4f12-80b1-14d1992bedf5" (UID: "41302545-fcf3-4f12-80b1-14d1992bedf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:51.665391 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.665315 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-workload-socket\") pod \"41302545-fcf3-4f12-80b1-14d1992bedf5\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " Oct 30 02:04:51.665391 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.665341 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-workload-certs\") pod \"41302545-fcf3-4f12-80b1-14d1992bedf5\" (UID: \"41302545-fcf3-4f12-80b1-14d1992bedf5\") " Oct 30 02:04:51.665591 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.665480 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-data" pod "41302545-fcf3-4f12-80b1-14d1992bedf5" (UID: "41302545-fcf3-4f12-80b1-14d1992bedf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:51.665977 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.665902 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-workload-certs" pod "41302545-fcf3-4f12-80b1-14d1992bedf5" (UID: "41302545-fcf3-4f12-80b1-14d1992bedf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:51.666127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.666108 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-workload-socket" pod "41302545-fcf3-4f12-80b1-14d1992bedf5" (UID: "41302545-fcf3-4f12-80b1-14d1992bedf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:51.667006 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.666977 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.667127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.667009 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.667127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.667024 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.667127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.667038 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.667692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.667670 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41302545-fcf3-4f12-80b1-14d1992bedf5-istiod-ca-cert" pod "41302545-fcf3-4f12-80b1-14d1992bedf5" (UID: "41302545-fcf3-4f12-80b1-14d1992bedf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:51.668297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.668272 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-podinfo" pod "41302545-fcf3-4f12-80b1-14d1992bedf5" (UID: "41302545-fcf3-4f12-80b1-14d1992bedf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:04:51.674121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.673260 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-envoy" pod "41302545-fcf3-4f12-80b1-14d1992bedf5" (UID: "41302545-fcf3-4f12-80b1-14d1992bedf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:51.674121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.673273 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41302545-fcf3-4f12-80b1-14d1992bedf5-kube-api-access-4hpxw" pod "41302545-fcf3-4f12-80b1-14d1992bedf5" (UID: "41302545-fcf3-4f12-80b1-14d1992bedf5"). InnerVolumeSpecName "kube-api-access-4hpxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:51.674121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.674047 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-token" pod "41302545-fcf3-4f12-80b1-14d1992bedf5" (UID: "41302545-fcf3-4f12-80b1-14d1992bedf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:51.768110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.768067 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.768259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.768121 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hpxw\" (UniqueName: \"kubernetes.io/projected/41302545-fcf3-4f12-80b1-14d1992bedf5-kube-api-access-4hpxw\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.768259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.768133 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.768259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.768142 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41302545-fcf3-4f12-80b1-14d1992bedf5-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:51.768259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:51.768150 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41302545-fcf3-4f12-80b1-14d1992bedf5-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:04:52.218435 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:52.218187 5998 generic.go:356] "Generic (PLEG): container finished" podID="41302545-fcf3-4f12-80b1-14d1992bedf5" containerID="a6b7c49602c455bafdd28b55ddc97d0e9e274726f3d236f68a424b26cff1f24a" exitCode=0 Oct 30 02:04:52.218435 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:52.218223 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" event={"ID":"41302545-fcf3-4f12-80b1-14d1992bedf5","Type":"ContainerDied","Data":"a6b7c49602c455bafdd28b55ddc97d0e9e274726f3d236f68a424b26cff1f24a"} Oct 30 02:04:52.218435 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:52.218261 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" Oct 30 02:04:52.218435 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:52.218289 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n" event={"ID":"41302545-fcf3-4f12-80b1-14d1992bedf5","Type":"ContainerDied","Data":"bc2a587658a811036dbd666b7a95a10169cb5e6cdecdcb8b34cbe638d93fe114"} Oct 30 02:04:52.218435 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:52.218317 5998 scope.go:122] "RemoveContainer" containerID="a6b7c49602c455bafdd28b55ddc97d0e9e274726f3d236f68a424b26cff1f24a" Oct 30 02:04:52.244562 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:52.244536 5998 scope.go:122] "RemoveContainer" containerID="a6b7c49602c455bafdd28b55ddc97d0e9e274726f3d236f68a424b26cff1f24a" Oct 30 02:04:52.245042 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:04:52.245017 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b7c49602c455bafdd28b55ddc97d0e9e274726f3d236f68a424b26cff1f24a\": container with ID starting with a6b7c49602c455bafdd28b55ddc97d0e9e274726f3d236f68a424b26cff1f24a not found: ID does not exist" containerID="a6b7c49602c455bafdd28b55ddc97d0e9e274726f3d236f68a424b26cff1f24a" Oct 30 02:04:52.245186 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:52.245053 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b7c49602c455bafdd28b55ddc97d0e9e274726f3d236f68a424b26cff1f24a"} err="failed to get container status \"a6b7c49602c455bafdd28b55ddc97d0e9e274726f3d236f68a424b26cff1f24a\": rpc error: code = NotFound desc = could not find container \"a6b7c49602c455bafdd28b55ddc97d0e9e274726f3d236f68a424b26cff1f24a\": container with ID starting with a6b7c49602c455bafdd28b55ddc97d0e9e274726f3d236f68a424b26cff1f24a not found: ID does not exist" Oct 30 02:04:52.265882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:52.265850 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n"] Oct 30 02:04:52.276861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:52.276835 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-70a069-openshift-default-c467dfd9c-zgl9n"] Oct 30 02:04:52.542809 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:52.542729 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41302545-fcf3-4f12-80b1-14d1992bedf5" path="/var/lib/kubelet/pods/41302545-fcf3-4f12-80b1-14d1992bedf5/volumes" Oct 30 02:04:52.625715 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:52.625674 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:53.243109 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:53.234267 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:53.355570 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:53.351823 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:53.877860 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:53.877829 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:54.502589 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:54.502555 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:55.218965 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:04:55.218914 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:04:55.283459 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:55.283427 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:56.783264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:56.783222 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:57.175317 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:57.175279 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25"] Oct 30 02:04:57.175521 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:57.175484 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" podUID="43cfa231-6176-4512-9307-aa872cbd3f6a" containerName="istio-proxy" containerID="cri-o://126ec86eb84a33e929617f6a64ff358416602e33ddb9f14103ce4f5038173e70" gracePeriod=30 Oct 30 02:04:57.283226 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:57.283189 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:04:58.516334 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:58.514798 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:04:58.754728 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:04:58.754680 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:00.301999 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.301966 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:00.422290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.422255 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk"] Oct 30 02:05:00.422798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.422782 5998 cpu_manager.go:414] "containerMap: removing container" podUID="321d663c-9621-4376-9ef9-f8aeeafa3f20" containerName="istio-proxy" Oct 30 02:05:00.422798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.422797 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321d663c-9621-4376-9ef9-f8aeeafa3f20" containerName="istio-proxy" Oct 30 02:05:00.422936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.422806 5998 cpu_manager.go:414] "containerMap: removing container" podUID="41302545-fcf3-4f12-80b1-14d1992bedf5" containerName="istio-proxy" Oct 30 02:05:00.422936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.422811 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41302545-fcf3-4f12-80b1-14d1992bedf5" containerName="istio-proxy" Oct 30 02:05:00.422936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.422823 5998 cpu_manager.go:414] "containerMap: removing container" podUID="eb00c574-0b03-48f4-a555-117ed93a9159" containerName="revoke-keys" Oct 30 02:05:00.422936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.422828 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb00c574-0b03-48f4-a555-117ed93a9159" containerName="revoke-keys" Oct 30 02:05:00.422936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.422919 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="321d663c-9621-4376-9ef9-f8aeeafa3f20" containerName="istio-proxy" Oct 30 02:05:00.422936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.422926 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="41302545-fcf3-4f12-80b1-14d1992bedf5" containerName="istio-proxy" Oct 30 02:05:00.422936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.422933 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb00c574-0b03-48f4-a555-117ed93a9159" containerName="revoke-keys" Oct 30 02:05:00.426328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.426307 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk" Oct 30 02:05:00.439958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.439930 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:05:00.450013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.449977 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:05:00.450180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.450108 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:05:00.450180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.450142 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:05:00.453800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.453773 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dded0acd-54b7-4148-a80a-f604e7722217-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3955582b-tnglk\" (UID: \"dded0acd-54b7-4148-a80a-f604e7722217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk" Oct 30 02:05:00.453894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.453814 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dded0acd-54b7-4148-a80a-f604e7722217-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3955582b-tnglk\" (UID: \"dded0acd-54b7-4148-a80a-f604e7722217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk" Oct 30 02:05:00.454147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.454084 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmm6d\" (UniqueName: \"kubernetes.io/projected/dded0acd-54b7-4148-a80a-f604e7722217-kube-api-access-fmm6d\") pod \"maas-api-revoke-sub-3955582b-tnglk\" (UID: \"dded0acd-54b7-4148-a80a-f604e7722217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk" Oct 30 02:05:00.454350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.454125 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk"] Oct 30 02:05:00.459817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.459783 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:00.555351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.555260 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dded0acd-54b7-4148-a80a-f604e7722217-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3955582b-tnglk\" (UID: \"dded0acd-54b7-4148-a80a-f604e7722217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk" Oct 30 02:05:00.555351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.555303 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dded0acd-54b7-4148-a80a-f604e7722217-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3955582b-tnglk\" (UID: \"dded0acd-54b7-4148-a80a-f604e7722217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk" Oct 30 02:05:00.555601 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.555484 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmm6d\" (UniqueName: \"kubernetes.io/projected/dded0acd-54b7-4148-a80a-f604e7722217-kube-api-access-fmm6d\") pod \"maas-api-revoke-sub-3955582b-tnglk\" (UID: \"dded0acd-54b7-4148-a80a-f604e7722217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk" Oct 30 02:05:00.556101 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.556067 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dded0acd-54b7-4148-a80a-f604e7722217-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3955582b-tnglk\" (UID: \"dded0acd-54b7-4148-a80a-f604e7722217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk" Oct 30 02:05:00.557879 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.557855 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dded0acd-54b7-4148-a80a-f604e7722217-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3955582b-tnglk\" (UID: \"dded0acd-54b7-4148-a80a-f604e7722217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk" Oct 30 02:05:00.580797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.580770 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmm6d\" (UniqueName: \"kubernetes.io/projected/dded0acd-54b7-4148-a80a-f604e7722217-kube-api-access-fmm6d\") pod \"maas-api-revoke-sub-3955582b-tnglk\" (UID: \"dded0acd-54b7-4148-a80a-f604e7722217\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk" Oct 30 02:05:00.603911 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.603879 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:00.740988 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:00.740953 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk" Oct 30 02:05:01.047501 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:01.047455 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk"] Oct 30 02:05:01.051056 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:05:01.051027 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddded0acd_54b7_4148_a80a_f604e7722217.slice/crio-76ecdb2b23e5bda1142c5ec509afc38b2343bacd684b0d31de3a3499096cb961 WatchSource:0}: Error finding container 76ecdb2b23e5bda1142c5ec509afc38b2343bacd684b0d31de3a3499096cb961: Status 404 returned error can't find the container with id 76ecdb2b23e5bda1142c5ec509afc38b2343bacd684b0d31de3a3499096cb961 Oct 30 02:05:01.119699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:01.119665 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:01.342854 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:01.342818 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk" event={"ID":"dded0acd-54b7-4148-a80a-f604e7722217","Type":"ContainerStarted","Data":"8a42562e87ae64fcced8b51cd9823f81bdc17241fa097e6e3638b02999a034d6"} Oct 30 02:05:01.343360 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:01.342881 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk" event={"ID":"dded0acd-54b7-4148-a80a-f604e7722217","Type":"ContainerStarted","Data":"76ecdb2b23e5bda1142c5ec509afc38b2343bacd684b0d31de3a3499096cb961"} Oct 30 02:05:01.384354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:01.384295 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk" podStartSLOduration=1.384279453 podStartE2EDuration="1.384279453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:01.379664687 +0000 UTC m=+2130.958455768" watchObservedRunningTime="2026-10-30 02:05:01.384279453 +0000 UTC m=+2130.963070534" Oct 30 02:05:01.500784 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:01.500747 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:02.358522 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.358481 5998 generic.go:356] "Generic (PLEG): container finished" podID="dded0acd-54b7-4148-a80a-f604e7722217" containerID="8a42562e87ae64fcced8b51cd9823f81bdc17241fa097e6e3638b02999a034d6" exitCode=0 Oct 30 02:05:02.358982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.358591 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk" event={"ID":"dded0acd-54b7-4148-a80a-f604e7722217","Type":"ContainerDied","Data":"8a42562e87ae64fcced8b51cd9823f81bdc17241fa097e6e3638b02999a034d6"} Oct 30 02:05:02.361084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.361053 5998 generic.go:356] "Generic (PLEG): container finished" podID="43cfa231-6176-4512-9307-aa872cbd3f6a" containerID="126ec86eb84a33e929617f6a64ff358416602e33ddb9f14103ce4f5038173e70" exitCode=0 Oct 30 02:05:02.361295 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.361274 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" event={"ID":"43cfa231-6176-4512-9307-aa872cbd3f6a","Type":"ContainerDied","Data":"126ec86eb84a33e929617f6a64ff358416602e33ddb9f14103ce4f5038173e70"} Oct 30 02:05:02.461167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.461060 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 02:05:02.500436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.500398 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-envoy\") pod \"43cfa231-6176-4512-9307-aa872cbd3f6a\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " Oct 30 02:05:02.500611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.500492 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-token\" (UniqueName: \"kubernetes.io/projected/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-token\") pod \"43cfa231-6176-4512-9307-aa872cbd3f6a\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " Oct 30 02:05:02.500611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.500527 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-podinfo\") pod \"43cfa231-6176-4512-9307-aa872cbd3f6a\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " Oct 30 02:05:02.500611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.500590 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-workload-socket\") pod \"43cfa231-6176-4512-9307-aa872cbd3f6a\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " Oct 30 02:05:02.500783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.500641 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-data\") pod \"43cfa231-6176-4512-9307-aa872cbd3f6a\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " Oct 30 02:05:02.500783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.500674 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43cfa231-6176-4512-9307-aa872cbd3f6a-kube-api-access-6l9jv\" (UniqueName: \"kubernetes.io/projected/43cfa231-6176-4512-9307-aa872cbd3f6a-kube-api-access-6l9jv\") pod \"43cfa231-6176-4512-9307-aa872cbd3f6a\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " Oct 30 02:05:02.500783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.500725 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-credential-socket\") pod \"43cfa231-6176-4512-9307-aa872cbd3f6a\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " Oct 30 02:05:02.500783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.500776 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43cfa231-6176-4512-9307-aa872cbd3f6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cfa231-6176-4512-9307-aa872cbd3f6a-istiod-ca-cert\") pod \"43cfa231-6176-4512-9307-aa872cbd3f6a\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " Oct 30 02:05:02.500972 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.500872 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-workload-certs\") pod \"43cfa231-6176-4512-9307-aa872cbd3f6a\" (UID: \"43cfa231-6176-4512-9307-aa872cbd3f6a\") " Oct 30 02:05:02.501030 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.500995 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-data" pod "43cfa231-6176-4512-9307-aa872cbd3f6a" (UID: "43cfa231-6176-4512-9307-aa872cbd3f6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:02.501148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.501082 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-credential-socket" pod "43cfa231-6176-4512-9307-aa872cbd3f6a" (UID: "43cfa231-6176-4512-9307-aa872cbd3f6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:02.501333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.501255 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-workload-socket" pod "43cfa231-6176-4512-9307-aa872cbd3f6a" (UID: "43cfa231-6176-4512-9307-aa872cbd3f6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:02.501401 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.501365 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-workload-certs" pod "43cfa231-6176-4512-9307-aa872cbd3f6a" (UID: "43cfa231-6176-4512-9307-aa872cbd3f6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:02.501474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.501395 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43cfa231-6176-4512-9307-aa872cbd3f6a-istiod-ca-cert" pod "43cfa231-6176-4512-9307-aa872cbd3f6a" (UID: "43cfa231-6176-4512-9307-aa872cbd3f6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:02.501617 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.501599 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:02.501686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.501618 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:02.501686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.501634 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cfa231-6176-4512-9307-aa872cbd3f6a-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:02.501686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.501648 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:02.501686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.501666 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:02.503423 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.503394 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cfa231-6176-4512-9307-aa872cbd3f6a-kube-api-access-6l9jv" pod "43cfa231-6176-4512-9307-aa872cbd3f6a" (UID: "43cfa231-6176-4512-9307-aa872cbd3f6a"). InnerVolumeSpecName "kube-api-access-6l9jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:02.504078 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.504060 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-podinfo" pod "43cfa231-6176-4512-9307-aa872cbd3f6a" (UID: "43cfa231-6176-4512-9307-aa872cbd3f6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:05:02.504173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.504132 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-token" pod "43cfa231-6176-4512-9307-aa872cbd3f6a" (UID: "43cfa231-6176-4512-9307-aa872cbd3f6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:02.504352 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.504333 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-envoy" pod "43cfa231-6176-4512-9307-aa872cbd3f6a" (UID: "43cfa231-6176-4512-9307-aa872cbd3f6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:02.565006 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.564967 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:02.603199 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.603157 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:02.603199 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.603191 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:02.603199 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.603200 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l9jv\" (UniqueName: \"kubernetes.io/projected/43cfa231-6176-4512-9307-aa872cbd3f6a-kube-api-access-6l9jv\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:02.603421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:02.603209 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cfa231-6176-4512-9307-aa872cbd3f6a-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:03.370871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:03.370843 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" Oct 30 02:05:03.371372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:03.370873 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25" event={"ID":"43cfa231-6176-4512-9307-aa872cbd3f6a","Type":"ContainerDied","Data":"e28bf9786859c97f1cd53db0e9329e739d50a9e56f1b7f73279333e69db22194"} Oct 30 02:05:03.371372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:03.370942 5998 scope.go:122] "RemoveContainer" containerID="126ec86eb84a33e929617f6a64ff358416602e33ddb9f14103ce4f5038173e70" Oct 30 02:05:03.419369 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:03.419334 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25"] Oct 30 02:05:03.431839 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:03.431805 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-kmr25"] Oct 30 02:05:03.553514 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:03.553483 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk" Oct 30 02:05:03.616898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:03.616851 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dded0acd-54b7-4148-a80a-f604e7722217-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dded0acd-54b7-4148-a80a-f604e7722217-maas-api-openshift-ca\") pod \"dded0acd-54b7-4148-a80a-f604e7722217\" (UID: \"dded0acd-54b7-4148-a80a-f604e7722217\") " Oct 30 02:05:03.617153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:03.617134 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dded0acd-54b7-4148-a80a-f604e7722217-kube-api-access-fmm6d\" (UniqueName: \"kubernetes.io/projected/dded0acd-54b7-4148-a80a-f604e7722217-kube-api-access-fmm6d\") pod \"dded0acd-54b7-4148-a80a-f604e7722217\" (UID: \"dded0acd-54b7-4148-a80a-f604e7722217\") " Oct 30 02:05:03.617259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:03.617227 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dded0acd-54b7-4148-a80a-f604e7722217-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dded0acd-54b7-4148-a80a-f604e7722217-maas-api-xks-ca\") pod \"dded0acd-54b7-4148-a80a-f604e7722217\" (UID: \"dded0acd-54b7-4148-a80a-f604e7722217\") " Oct 30 02:05:03.617315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:03.617237 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dded0acd-54b7-4148-a80a-f604e7722217-maas-api-openshift-ca" pod "dded0acd-54b7-4148-a80a-f604e7722217" (UID: "dded0acd-54b7-4148-a80a-f604e7722217"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:03.617678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:03.617656 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dded0acd-54b7-4148-a80a-f604e7722217-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:03.619894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:03.619862 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dded0acd-54b7-4148-a80a-f604e7722217-kube-api-access-fmm6d" pod "dded0acd-54b7-4148-a80a-f604e7722217" (UID: "dded0acd-54b7-4148-a80a-f604e7722217"). InnerVolumeSpecName "kube-api-access-fmm6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:03.620024 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:03.619874 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dded0acd-54b7-4148-a80a-f604e7722217-maas-api-xks-ca" pod "dded0acd-54b7-4148-a80a-f604e7722217" (UID: "dded0acd-54b7-4148-a80a-f604e7722217"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:03.719043 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:03.718988 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmm6d\" (UniqueName: \"kubernetes.io/projected/dded0acd-54b7-4148-a80a-f604e7722217-kube-api-access-fmm6d\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:03.719043 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:03.719037 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dded0acd-54b7-4148-a80a-f604e7722217-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:03.844437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:03.844398 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:04.242799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.242760 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n"] Oct 30 02:05:04.243552 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.243528 5998 cpu_manager.go:414] "containerMap: removing container" podUID="dded0acd-54b7-4148-a80a-f604e7722217" containerName="revoke-keys" Oct 30 02:05:04.243639 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.243555 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dded0acd-54b7-4148-a80a-f604e7722217" containerName="revoke-keys" Oct 30 02:05:04.243639 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.243582 5998 cpu_manager.go:414] "containerMap: removing container" podUID="43cfa231-6176-4512-9307-aa872cbd3f6a" containerName="istio-proxy" Oct 30 02:05:04.243639 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.243591 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43cfa231-6176-4512-9307-aa872cbd3f6a" containerName="istio-proxy" Oct 30 02:05:04.243892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.243873 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="43cfa231-6176-4512-9307-aa872cbd3f6a" containerName="istio-proxy" Oct 30 02:05:04.243952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.243911 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="dded0acd-54b7-4148-a80a-f604e7722217" containerName="revoke-keys" Oct 30 02:05:04.248000 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.247974 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n" Oct 30 02:05:04.267406 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.267375 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n"] Oct 30 02:05:04.342935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.342892 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/225888f1-1466-4e02-a0e0-4a34e7ae1c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2acf668b-sts4n\" (UID: \"225888f1-1466-4e02-a0e0-4a34e7ae1c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n" Oct 30 02:05:04.343183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.343011 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/225888f1-1466-4e02-a0e0-4a34e7ae1c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2acf668b-sts4n\" (UID: \"225888f1-1466-4e02-a0e0-4a34e7ae1c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n" Oct 30 02:05:04.343183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.343048 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv4ng\" (UniqueName: \"kubernetes.io/projected/225888f1-1466-4e02-a0e0-4a34e7ae1c28-kube-api-access-kv4ng\") pod \"maas-api-revoke-sub-2acf668b-sts4n\" (UID: \"225888f1-1466-4e02-a0e0-4a34e7ae1c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n" Oct 30 02:05:04.374336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.374302 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:04.382065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.382035 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk" Oct 30 02:05:04.382456 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.382038 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk" event={"ID":"dded0acd-54b7-4148-a80a-f604e7722217","Type":"ContainerDied","Data":"76ecdb2b23e5bda1142c5ec509afc38b2343bacd684b0d31de3a3499096cb961"} Oct 30 02:05:04.382586 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.382494 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76ecdb2b23e5bda1142c5ec509afc38b2343bacd684b0d31de3a3499096cb961" Oct 30 02:05:04.444338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.444290 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/225888f1-1466-4e02-a0e0-4a34e7ae1c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2acf668b-sts4n\" (UID: \"225888f1-1466-4e02-a0e0-4a34e7ae1c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n" Oct 30 02:05:04.444555 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.444427 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/225888f1-1466-4e02-a0e0-4a34e7ae1c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2acf668b-sts4n\" (UID: \"225888f1-1466-4e02-a0e0-4a34e7ae1c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n" Oct 30 02:05:04.444555 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.444480 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4ng\" (UniqueName: \"kubernetes.io/projected/225888f1-1466-4e02-a0e0-4a34e7ae1c28-kube-api-access-kv4ng\") pod \"maas-api-revoke-sub-2acf668b-sts4n\" (UID: \"225888f1-1466-4e02-a0e0-4a34e7ae1c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n" Oct 30 02:05:04.447126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.445870 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/225888f1-1466-4e02-a0e0-4a34e7ae1c28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2acf668b-sts4n\" (UID: \"225888f1-1466-4e02-a0e0-4a34e7ae1c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n" Oct 30 02:05:04.452773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.452740 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/225888f1-1466-4e02-a0e0-4a34e7ae1c28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2acf668b-sts4n\" (UID: \"225888f1-1466-4e02-a0e0-4a34e7ae1c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n" Oct 30 02:05:04.466821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.466793 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv4ng\" (UniqueName: \"kubernetes.io/projected/225888f1-1466-4e02-a0e0-4a34e7ae1c28-kube-api-access-kv4ng\") pod \"maas-api-revoke-sub-2acf668b-sts4n\" (UID: \"225888f1-1466-4e02-a0e0-4a34e7ae1c28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n" Oct 30 02:05:04.543371 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.543277 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43cfa231-6176-4512-9307-aa872cbd3f6a" path="/var/lib/kubelet/pods/43cfa231-6176-4512-9307-aa872cbd3f6a/volumes" Oct 30 02:05:04.562254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.562221 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n" Oct 30 02:05:04.888305 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:04.887672 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n"] Oct 30 02:05:04.890660 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:05:04.890598 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225888f1_1466_4e02_a0e0_4a34e7ae1c28.slice/crio-5318a47b4bcb6c0b143119572a915545ee5538c228180534387b7a52a3bbce91 WatchSource:0}: Error finding container 5318a47b4bcb6c0b143119572a915545ee5538c228180534387b7a52a3bbce91: Status 404 returned error can't find the container with id 5318a47b4bcb6c0b143119572a915545ee5538c228180534387b7a52a3bbce91 Oct 30 02:05:05.391770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:05.391733 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n" event={"ID":"225888f1-1466-4e02-a0e0-4a34e7ae1c28","Type":"ContainerStarted","Data":"99ebdc01b16d1b695cfd53d7b5b0dbb2ad5cc93709762df201381c0701131e7c"} Oct 30 02:05:05.392213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:05.391786 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n" event={"ID":"225888f1-1466-4e02-a0e0-4a34e7ae1c28","Type":"ContainerStarted","Data":"5318a47b4bcb6c0b143119572a915545ee5538c228180534387b7a52a3bbce91"} Oct 30 02:05:05.430718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:05.430661 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n" podStartSLOduration=1.4306458100000001 podStartE2EDuration="1.43064581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:05.428936591 +0000 UTC m=+2135.007727671" watchObservedRunningTime="2026-10-30 02:05:05.43064581 +0000 UTC m=+2135.009436938" Oct 30 02:05:06.161961 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:06.161926 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:06.400082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:06.400045 5998 generic.go:356] "Generic (PLEG): container finished" podID="225888f1-1466-4e02-a0e0-4a34e7ae1c28" containerID="99ebdc01b16d1b695cfd53d7b5b0dbb2ad5cc93709762df201381c0701131e7c" exitCode=0 Oct 30 02:05:06.400565 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:06.400127 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n" event={"ID":"225888f1-1466-4e02-a0e0-4a34e7ae1c28","Type":"ContainerDied","Data":"99ebdc01b16d1b695cfd53d7b5b0dbb2ad5cc93709762df201381c0701131e7c"} Oct 30 02:05:06.732283 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:06.732250 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:06.839979 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:06.839940 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:07.568863 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:07.568816 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n" Oct 30 02:05:07.690229 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:07.690193 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/225888f1-1466-4e02-a0e0-4a34e7ae1c28-kube-api-access-kv4ng\" (UniqueName: \"kubernetes.io/projected/225888f1-1466-4e02-a0e0-4a34e7ae1c28-kube-api-access-kv4ng\") pod \"225888f1-1466-4e02-a0e0-4a34e7ae1c28\" (UID: \"225888f1-1466-4e02-a0e0-4a34e7ae1c28\") " Oct 30 02:05:07.690446 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:07.690255 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/225888f1-1466-4e02-a0e0-4a34e7ae1c28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/225888f1-1466-4e02-a0e0-4a34e7ae1c28-maas-api-openshift-ca\") pod \"225888f1-1466-4e02-a0e0-4a34e7ae1c28\" (UID: \"225888f1-1466-4e02-a0e0-4a34e7ae1c28\") " Oct 30 02:05:07.690446 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:07.690369 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/225888f1-1466-4e02-a0e0-4a34e7ae1c28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/225888f1-1466-4e02-a0e0-4a34e7ae1c28-maas-api-xks-ca\") pod \"225888f1-1466-4e02-a0e0-4a34e7ae1c28\" (UID: \"225888f1-1466-4e02-a0e0-4a34e7ae1c28\") " Oct 30 02:05:07.690682 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:07.690658 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225888f1-1466-4e02-a0e0-4a34e7ae1c28-maas-api-openshift-ca" pod "225888f1-1466-4e02-a0e0-4a34e7ae1c28" (UID: "225888f1-1466-4e02-a0e0-4a34e7ae1c28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:07.690955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:07.690928 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/225888f1-1466-4e02-a0e0-4a34e7ae1c28-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:07.692371 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:07.692349 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/225888f1-1466-4e02-a0e0-4a34e7ae1c28-maas-api-xks-ca" pod "225888f1-1466-4e02-a0e0-4a34e7ae1c28" (UID: "225888f1-1466-4e02-a0e0-4a34e7ae1c28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:07.692458 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:07.692442 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225888f1-1466-4e02-a0e0-4a34e7ae1c28-kube-api-access-kv4ng" pod "225888f1-1466-4e02-a0e0-4a34e7ae1c28" (UID: "225888f1-1466-4e02-a0e0-4a34e7ae1c28"). InnerVolumeSpecName "kube-api-access-kv4ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:07.791996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:07.791954 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv4ng\" (UniqueName: \"kubernetes.io/projected/225888f1-1466-4e02-a0e0-4a34e7ae1c28-kube-api-access-kv4ng\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:07.791996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:07.791994 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/225888f1-1466-4e02-a0e0-4a34e7ae1c28-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:07.813923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:07.813888 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:08.416619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:08.416585 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n" Oct 30 02:05:08.416827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:08.416614 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n" event={"ID":"225888f1-1466-4e02-a0e0-4a34e7ae1c28","Type":"ContainerDied","Data":"5318a47b4bcb6c0b143119572a915545ee5538c228180534387b7a52a3bbce91"} Oct 30 02:05:08.416827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:08.416694 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5318a47b4bcb6c0b143119572a915545ee5538c228180534387b7a52a3bbce91" Oct 30 02:05:09.366515 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:09.366473 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:09.892304 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:09.892271 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:11.481589 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:11.481550 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:12.886135 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:12.883314 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:13.048112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.047835 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:13.154478 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.154375 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:13.187549 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.187308 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r"] Oct 30 02:05:13.188029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.188006 5998 cpu_manager.go:414] "containerMap: removing container" podUID="225888f1-1466-4e02-a0e0-4a34e7ae1c28" containerName="revoke-keys" Oct 30 02:05:13.188029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.188028 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="225888f1-1466-4e02-a0e0-4a34e7ae1c28" containerName="revoke-keys" Oct 30 02:05:13.188278 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.188162 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="225888f1-1466-4e02-a0e0-4a34e7ae1c28" containerName="revoke-keys" Oct 30 02:05:13.191719 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.191700 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r" Oct 30 02:05:13.217110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.211806 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r"] Oct 30 02:05:13.224111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.218032 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:05:13.224111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.218325 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:05:13.224111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.218443 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:05:13.224111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.219028 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:05:13.260030 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.259990 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvm9x\" (UniqueName: \"kubernetes.io/projected/93fb3715-c2f0-48dc-b89b-a98d27b49440-kube-api-access-xvm9x\") pod \"maas-api-revoke-sub-ce2f3aab-cwt5r\" (UID: \"93fb3715-c2f0-48dc-b89b-a98d27b49440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r" Oct 30 02:05:13.260235 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.260205 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fb3715-c2f0-48dc-b89b-a98d27b49440-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce2f3aab-cwt5r\" (UID: \"93fb3715-c2f0-48dc-b89b-a98d27b49440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r" Oct 30 02:05:13.260308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.260248 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fb3715-c2f0-48dc-b89b-a98d27b49440-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce2f3aab-cwt5r\" (UID: \"93fb3715-c2f0-48dc-b89b-a98d27b49440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r" Oct 30 02:05:13.484821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.482696 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fb3715-c2f0-48dc-b89b-a98d27b49440-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce2f3aab-cwt5r\" (UID: \"93fb3715-c2f0-48dc-b89b-a98d27b49440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r" Oct 30 02:05:13.484821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.482848 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvm9x\" (UniqueName: \"kubernetes.io/projected/93fb3715-c2f0-48dc-b89b-a98d27b49440-kube-api-access-xvm9x\") pod \"maas-api-revoke-sub-ce2f3aab-cwt5r\" (UID: \"93fb3715-c2f0-48dc-b89b-a98d27b49440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r" Oct 30 02:05:13.484821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.482908 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fb3715-c2f0-48dc-b89b-a98d27b49440-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce2f3aab-cwt5r\" (UID: \"93fb3715-c2f0-48dc-b89b-a98d27b49440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r" Oct 30 02:05:13.485606 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.485577 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fb3715-c2f0-48dc-b89b-a98d27b49440-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce2f3aab-cwt5r\" (UID: \"93fb3715-c2f0-48dc-b89b-a98d27b49440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r" Oct 30 02:05:13.486077 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.486055 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fb3715-c2f0-48dc-b89b-a98d27b49440-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce2f3aab-cwt5r\" (UID: \"93fb3715-c2f0-48dc-b89b-a98d27b49440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r" Oct 30 02:05:13.507645 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.507618 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvm9x\" (UniqueName: \"kubernetes.io/projected/93fb3715-c2f0-48dc-b89b-a98d27b49440-kube-api-access-xvm9x\") pod \"maas-api-revoke-sub-ce2f3aab-cwt5r\" (UID: \"93fb3715-c2f0-48dc-b89b-a98d27b49440\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r" Oct 30 02:05:13.748043 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.747956 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:13.805628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:13.805588 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r" Oct 30 02:05:14.122682 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:14.122593 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r"] Oct 30 02:05:14.131128 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:05:14.126633 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93fb3715_c2f0_48dc_b89b_a98d27b49440.slice/crio-01e6160afaf94318d1f7a134ee824bee060117d0cdab61bb77660e12e1166f0c WatchSource:0}: Error finding container 01e6160afaf94318d1f7a134ee824bee060117d0cdab61bb77660e12e1166f0c: Status 404 returned error can't find the container with id 01e6160afaf94318d1f7a134ee824bee060117d0cdab61bb77660e12e1166f0c Oct 30 02:05:14.479264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:14.479209 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r" event={"ID":"93fb3715-c2f0-48dc-b89b-a98d27b49440","Type":"ContainerStarted","Data":"1af10d8dfc3909b3eb2a99260f87be91a0be4d56b7284f91ba90ebdeccf2565f"} Oct 30 02:05:14.479453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:14.479323 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r" event={"ID":"93fb3715-c2f0-48dc-b89b-a98d27b49440","Type":"ContainerStarted","Data":"01e6160afaf94318d1f7a134ee824bee060117d0cdab61bb77660e12e1166f0c"} Oct 30 02:05:14.518725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:14.518657 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r" podStartSLOduration=1.518640364 podStartE2EDuration="1.518640364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:14.515822234 +0000 UTC m=+2144.094613328" watchObservedRunningTime="2026-10-30 02:05:14.518640364 +0000 UTC m=+2144.097431446" Oct 30 02:05:14.563518 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:14.563481 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr"] Oct 30 02:05:14.568338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:14.568311 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr" Oct 30 02:05:14.588348 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:14.588317 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr"] Oct 30 02:05:14.701063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:14.701020 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bhq4\" (UniqueName: \"kubernetes.io/projected/eefe0f81-82c8-4491-90b9-09448b23b850-kube-api-access-9bhq4\") pod \"maas-api-revoke-sub-b4de44e8-mjwlr\" (UID: \"eefe0f81-82c8-4491-90b9-09448b23b850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr" Oct 30 02:05:14.701284 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:14.701234 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefe0f81-82c8-4491-90b9-09448b23b850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4de44e8-mjwlr\" (UID: \"eefe0f81-82c8-4491-90b9-09448b23b850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr" Oct 30 02:05:14.701284 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:14.701260 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefe0f81-82c8-4491-90b9-09448b23b850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4de44e8-mjwlr\" (UID: \"eefe0f81-82c8-4491-90b9-09448b23b850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr" Oct 30 02:05:14.802816 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:14.802767 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefe0f81-82c8-4491-90b9-09448b23b850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4de44e8-mjwlr\" (UID: \"eefe0f81-82c8-4491-90b9-09448b23b850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr" Oct 30 02:05:14.802816 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:14.802820 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefe0f81-82c8-4491-90b9-09448b23b850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4de44e8-mjwlr\" (UID: \"eefe0f81-82c8-4491-90b9-09448b23b850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr" Oct 30 02:05:14.803070 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:14.802969 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bhq4\" (UniqueName: \"kubernetes.io/projected/eefe0f81-82c8-4491-90b9-09448b23b850-kube-api-access-9bhq4\") pod \"maas-api-revoke-sub-b4de44e8-mjwlr\" (UID: \"eefe0f81-82c8-4491-90b9-09448b23b850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr" Oct 30 02:05:14.803633 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:14.803599 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefe0f81-82c8-4491-90b9-09448b23b850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4de44e8-mjwlr\" (UID: \"eefe0f81-82c8-4491-90b9-09448b23b850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr" Oct 30 02:05:14.805514 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:14.805489 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefe0f81-82c8-4491-90b9-09448b23b850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4de44e8-mjwlr\" (UID: \"eefe0f81-82c8-4491-90b9-09448b23b850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr" Oct 30 02:05:14.825491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:14.825461 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bhq4\" (UniqueName: \"kubernetes.io/projected/eefe0f81-82c8-4491-90b9-09448b23b850-kube-api-access-9bhq4\") pod \"maas-api-revoke-sub-b4de44e8-mjwlr\" (UID: \"eefe0f81-82c8-4491-90b9-09448b23b850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr" Oct 30 02:05:14.883698 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:14.883665 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr" Oct 30 02:05:15.198891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:15.198859 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr"] Oct 30 02:05:15.202635 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:05:15.202599 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeefe0f81_82c8_4491_90b9_09448b23b850.slice/crio-d431393bc38a0b82523cf4d015e4d006022ea1760ea37f76c522be307b32efba WatchSource:0}: Error finding container d431393bc38a0b82523cf4d015e4d006022ea1760ea37f76c522be307b32efba: Status 404 returned error can't find the container with id d431393bc38a0b82523cf4d015e4d006022ea1760ea37f76c522be307b32efba Oct 30 02:05:15.486828 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:15.486728 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr" event={"ID":"eefe0f81-82c8-4491-90b9-09448b23b850","Type":"ContainerStarted","Data":"d431393bc38a0b82523cf4d015e4d006022ea1760ea37f76c522be307b32efba"} Oct 30 02:05:15.488407 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:15.488376 5998 generic.go:356] "Generic (PLEG): container finished" podID="93fb3715-c2f0-48dc-b89b-a98d27b49440" containerID="1af10d8dfc3909b3eb2a99260f87be91a0be4d56b7284f91ba90ebdeccf2565f" exitCode=0 Oct 30 02:05:15.488558 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:15.488458 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r" event={"ID":"93fb3715-c2f0-48dc-b89b-a98d27b49440","Type":"ContainerDied","Data":"1af10d8dfc3909b3eb2a99260f87be91a0be4d56b7284f91ba90ebdeccf2565f"} Oct 30 02:05:15.895404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:15.895317 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:16.259978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:16.259944 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:16.382997 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:16.382968 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:16.496202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:16.496115 5998 generic.go:356] "Generic (PLEG): container finished" podID="eefe0f81-82c8-4491-90b9-09448b23b850" containerID="89c2320aec011e56fa5b2a95da3713dded5c0fc3ba35b64e6bdebe070fcdc5e1" exitCode=0 Oct 30 02:05:16.496359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:16.496134 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr" event={"ID":"eefe0f81-82c8-4491-90b9-09448b23b850","Type":"ContainerDied","Data":"89c2320aec011e56fa5b2a95da3713dded5c0fc3ba35b64e6bdebe070fcdc5e1"} Oct 30 02:05:16.549270 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:16.549237 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:16.672790 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:16.672761 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r" Oct 30 02:05:16.835614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:16.835526 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93fb3715-c2f0-48dc-b89b-a98d27b49440-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fb3715-c2f0-48dc-b89b-a98d27b49440-maas-api-openshift-ca\") pod \"93fb3715-c2f0-48dc-b89b-a98d27b49440\" (UID: \"93fb3715-c2f0-48dc-b89b-a98d27b49440\") " Oct 30 02:05:16.835614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:16.835603 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93fb3715-c2f0-48dc-b89b-a98d27b49440-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fb3715-c2f0-48dc-b89b-a98d27b49440-maas-api-xks-ca\") pod \"93fb3715-c2f0-48dc-b89b-a98d27b49440\" (UID: \"93fb3715-c2f0-48dc-b89b-a98d27b49440\") " Oct 30 02:05:16.835867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:16.835757 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93fb3715-c2f0-48dc-b89b-a98d27b49440-kube-api-access-xvm9x\" (UniqueName: \"kubernetes.io/projected/93fb3715-c2f0-48dc-b89b-a98d27b49440-kube-api-access-xvm9x\") pod \"93fb3715-c2f0-48dc-b89b-a98d27b49440\" (UID: \"93fb3715-c2f0-48dc-b89b-a98d27b49440\") " Oct 30 02:05:16.836023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:16.835985 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93fb3715-c2f0-48dc-b89b-a98d27b49440-maas-api-openshift-ca" pod "93fb3715-c2f0-48dc-b89b-a98d27b49440" (UID: "93fb3715-c2f0-48dc-b89b-a98d27b49440"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:16.836439 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:16.836414 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fb3715-c2f0-48dc-b89b-a98d27b49440-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:16.838364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:16.838339 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93fb3715-c2f0-48dc-b89b-a98d27b49440-maas-api-xks-ca" pod "93fb3715-c2f0-48dc-b89b-a98d27b49440" (UID: "93fb3715-c2f0-48dc-b89b-a98d27b49440"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:16.838534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:16.838410 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93fb3715-c2f0-48dc-b89b-a98d27b49440-kube-api-access-xvm9x" pod "93fb3715-c2f0-48dc-b89b-a98d27b49440" (UID: "93fb3715-c2f0-48dc-b89b-a98d27b49440"). InnerVolumeSpecName "kube-api-access-xvm9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:16.938071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:16.938033 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fb3715-c2f0-48dc-b89b-a98d27b49440-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:16.938071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:16.938065 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvm9x\" (UniqueName: \"kubernetes.io/projected/93fb3715-c2f0-48dc-b89b-a98d27b49440-kube-api-access-xvm9x\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:16.959646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:16.959615 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:17.334797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:17.334757 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf"] Oct 30 02:05:17.335404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:17.335036 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" podUID="f8e03b95-821f-456c-a435-831d0bfa52f0" containerName="istio-proxy" containerID="cri-o://c56a2860f48966a3356b49c3b13c68743ff64be9de71f58be01bd78e7e4ff846" gracePeriod=30 Oct 30 02:05:17.380072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:17.380029 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:17.515120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:17.515073 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r" Oct 30 02:05:17.518190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:17.515425 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r" event={"ID":"93fb3715-c2f0-48dc-b89b-a98d27b49440","Type":"ContainerDied","Data":"01e6160afaf94318d1f7a134ee824bee060117d0cdab61bb77660e12e1166f0c"} Oct 30 02:05:17.518190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:17.518156 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01e6160afaf94318d1f7a134ee824bee060117d0cdab61bb77660e12e1166f0c" Oct 30 02:05:17.693238 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:17.693211 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr" Oct 30 02:05:17.811019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:17.810973 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:17.857022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:17.856036 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eefe0f81-82c8-4491-90b9-09448b23b850-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefe0f81-82c8-4491-90b9-09448b23b850-maas-api-xks-ca\") pod \"eefe0f81-82c8-4491-90b9-09448b23b850\" (UID: \"eefe0f81-82c8-4491-90b9-09448b23b850\") " Oct 30 02:05:17.857022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:17.856662 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eefe0f81-82c8-4491-90b9-09448b23b850-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefe0f81-82c8-4491-90b9-09448b23b850-maas-api-openshift-ca\") pod \"eefe0f81-82c8-4491-90b9-09448b23b850\" (UID: \"eefe0f81-82c8-4491-90b9-09448b23b850\") " Oct 30 02:05:17.857022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:17.856743 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eefe0f81-82c8-4491-90b9-09448b23b850-kube-api-access-9bhq4\" (UniqueName: \"kubernetes.io/projected/eefe0f81-82c8-4491-90b9-09448b23b850-kube-api-access-9bhq4\") pod \"eefe0f81-82c8-4491-90b9-09448b23b850\" (UID: \"eefe0f81-82c8-4491-90b9-09448b23b850\") " Oct 30 02:05:17.864432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:17.862761 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eefe0f81-82c8-4491-90b9-09448b23b850-maas-api-openshift-ca" pod "eefe0f81-82c8-4491-90b9-09448b23b850" (UID: "eefe0f81-82c8-4491-90b9-09448b23b850"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:17.872236 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:17.872195 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefe0f81-82c8-4491-90b9-09448b23b850-kube-api-access-9bhq4" pod "eefe0f81-82c8-4491-90b9-09448b23b850" (UID: "eefe0f81-82c8-4491-90b9-09448b23b850"). InnerVolumeSpecName "kube-api-access-9bhq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:17.873071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:17.873028 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eefe0f81-82c8-4491-90b9-09448b23b850-maas-api-xks-ca" pod "eefe0f81-82c8-4491-90b9-09448b23b850" (UID: "eefe0f81-82c8-4491-90b9-09448b23b850"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:17.958836 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:17.958803 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eefe0f81-82c8-4491-90b9-09448b23b850-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:17.958836 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:17.958829 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bhq4\" (UniqueName: \"kubernetes.io/projected/eefe0f81-82c8-4491-90b9-09448b23b850-kube-api-access-9bhq4\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:17.959041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:17.958845 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eefe0f81-82c8-4491-90b9-09448b23b850-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:18.116979 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:18.116861 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:18.523765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:18.523736 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr" Oct 30 02:05:18.523765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:18.523753 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr" event={"ID":"eefe0f81-82c8-4491-90b9-09448b23b850","Type":"ContainerDied","Data":"d431393bc38a0b82523cf4d015e4d006022ea1760ea37f76c522be307b32efba"} Oct 30 02:05:18.524302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:18.523802 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d431393bc38a0b82523cf4d015e4d006022ea1760ea37f76c522be307b32efba" Oct 30 02:05:19.475470 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:19.475434 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:20.684218 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:20.684183 5998 patch_prober.go:29] interesting pod/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:05:20.684639 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:20.684247 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" podUID="f8e03b95-821f-456c-a435-831d0bfa52f0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:05:21.077826 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:21.077703 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:22.607828 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:22.607789 5998 generic.go:356] "Generic (PLEG): container finished" podID="f8e03b95-821f-456c-a435-831d0bfa52f0" containerID="c56a2860f48966a3356b49c3b13c68743ff64be9de71f58be01bd78e7e4ff846" exitCode=0 Oct 30 02:05:22.609430 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:22.609401 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" event={"ID":"f8e03b95-821f-456c-a435-831d0bfa52f0","Type":"ContainerDied","Data":"c56a2860f48966a3356b49c3b13c68743ff64be9de71f58be01bd78e7e4ff846"} Oct 30 02:05:22.609620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:22.609605 5998 scope.go:122] "RemoveContainer" containerID="9ca51b1842e73bbb0337773e16edb2615575f589a2c25aaaeaaa959531c4aa23" Oct 30 02:05:23.075537 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.075509 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 02:05:23.132941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.132831 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-token\" (UniqueName: \"kubernetes.io/projected/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-token\") pod \"f8e03b95-821f-456c-a435-831d0bfa52f0\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " Oct 30 02:05:23.132941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.132928 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-credential-socket\") pod \"f8e03b95-821f-456c-a435-831d0bfa52f0\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " Oct 30 02:05:23.133175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.132973 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-podinfo\") pod \"f8e03b95-821f-456c-a435-831d0bfa52f0\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " Oct 30 02:05:23.133428 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.133399 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-credential-socket" pod "f8e03b95-821f-456c-a435-831d0bfa52f0" (UID: "f8e03b95-821f-456c-a435-831d0bfa52f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:23.133540 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.133453 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-workload-certs\") pod \"f8e03b95-821f-456c-a435-831d0bfa52f0\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " Oct 30 02:05:23.133655 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.133638 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8e03b95-821f-456c-a435-831d0bfa52f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8e03b95-821f-456c-a435-831d0bfa52f0-istiod-ca-cert\") pod \"f8e03b95-821f-456c-a435-831d0bfa52f0\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " Oct 30 02:05:23.133723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.133691 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-workload-socket\") pod \"f8e03b95-821f-456c-a435-831d0bfa52f0\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " Oct 30 02:05:23.133723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.133716 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-envoy\") pod \"f8e03b95-821f-456c-a435-831d0bfa52f0\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " Oct 30 02:05:23.133881 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.133717 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-workload-certs" pod "f8e03b95-821f-456c-a435-831d0bfa52f0" (UID: "f8e03b95-821f-456c-a435-831d0bfa52f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:23.133881 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.133755 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-data\") pod \"f8e03b95-821f-456c-a435-831d0bfa52f0\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " Oct 30 02:05:23.133881 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.133824 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8e03b95-821f-456c-a435-831d0bfa52f0-kube-api-access-4hv6q\" (UniqueName: \"kubernetes.io/projected/f8e03b95-821f-456c-a435-831d0bfa52f0-kube-api-access-4hv6q\") pod \"f8e03b95-821f-456c-a435-831d0bfa52f0\" (UID: \"f8e03b95-821f-456c-a435-831d0bfa52f0\") " Oct 30 02:05:23.134033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.134009 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e03b95-821f-456c-a435-831d0bfa52f0-istiod-ca-cert" pod "f8e03b95-821f-456c-a435-831d0bfa52f0" (UID: "f8e03b95-821f-456c-a435-831d0bfa52f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:23.134454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.134431 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8e03b95-821f-456c-a435-831d0bfa52f0-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:23.134542 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.134457 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:23.134542 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.134472 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:23.134928 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.134720 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-data" pod "f8e03b95-821f-456c-a435-831d0bfa52f0" (UID: "f8e03b95-821f-456c-a435-831d0bfa52f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:23.134928 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.134892 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-workload-socket" pod "f8e03b95-821f-456c-a435-831d0bfa52f0" (UID: "f8e03b95-821f-456c-a435-831d0bfa52f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:23.135867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.135843 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-podinfo" pod "f8e03b95-821f-456c-a435-831d0bfa52f0" (UID: "f8e03b95-821f-456c-a435-831d0bfa52f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:05:23.136169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.136143 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-token" pod "f8e03b95-821f-456c-a435-831d0bfa52f0" (UID: "f8e03b95-821f-456c-a435-831d0bfa52f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:23.136925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.136903 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-envoy" pod "f8e03b95-821f-456c-a435-831d0bfa52f0" (UID: "f8e03b95-821f-456c-a435-831d0bfa52f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:23.137000 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.136971 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e03b95-821f-456c-a435-831d0bfa52f0-kube-api-access-4hv6q" pod "f8e03b95-821f-456c-a435-831d0bfa52f0" (UID: "f8e03b95-821f-456c-a435-831d0bfa52f0"). InnerVolumeSpecName "kube-api-access-4hv6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:23.235571 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.235526 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:23.235571 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.235562 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:23.235571 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.235576 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:23.235883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.235589 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hv6q\" (UniqueName: \"kubernetes.io/projected/f8e03b95-821f-456c-a435-831d0bfa52f0-kube-api-access-4hv6q\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:23.235883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.235602 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:23.235883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.235615 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8e03b95-821f-456c-a435-831d0bfa52f0-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:23.475133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.475073 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:23.565923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.565883 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6"] Oct 30 02:05:23.566442 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.566426 5998 cpu_manager.go:414] "containerMap: removing container" podUID="eefe0f81-82c8-4491-90b9-09448b23b850" containerName="revoke-keys" Oct 30 02:05:23.566494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.566443 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eefe0f81-82c8-4491-90b9-09448b23b850" containerName="revoke-keys" Oct 30 02:05:23.566494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.566463 5998 cpu_manager.go:414] "containerMap: removing container" podUID="93fb3715-c2f0-48dc-b89b-a98d27b49440" containerName="revoke-keys" Oct 30 02:05:23.566494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.566469 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93fb3715-c2f0-48dc-b89b-a98d27b49440" containerName="revoke-keys" Oct 30 02:05:23.566494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.566490 5998 cpu_manager.go:414] "containerMap: removing container" podUID="f8e03b95-821f-456c-a435-831d0bfa52f0" containerName="istio-proxy" Oct 30 02:05:23.566620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.566496 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e03b95-821f-456c-a435-831d0bfa52f0" containerName="istio-proxy" Oct 30 02:05:23.566620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.566506 5998 cpu_manager.go:414] "containerMap: removing container" podUID="f8e03b95-821f-456c-a435-831d0bfa52f0" containerName="istio-proxy" Oct 30 02:05:23.566620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.566512 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e03b95-821f-456c-a435-831d0bfa52f0" containerName="istio-proxy" Oct 30 02:05:23.566620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.566596 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e03b95-821f-456c-a435-831d0bfa52f0" containerName="istio-proxy" Oct 30 02:05:23.566620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.566606 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e03b95-821f-456c-a435-831d0bfa52f0" containerName="istio-proxy" Oct 30 02:05:23.566620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.566617 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="eefe0f81-82c8-4491-90b9-09448b23b850" containerName="revoke-keys" Oct 30 02:05:23.567191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.566625 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="93fb3715-c2f0-48dc-b89b-a98d27b49440" containerName="revoke-keys" Oct 30 02:05:23.570268 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.570252 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6" Oct 30 02:05:23.583310 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.583278 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:05:23.583531 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.583514 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:05:23.583616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.583599 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:05:23.583674 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.583634 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:05:23.602282 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.599179 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6"] Oct 30 02:05:23.617531 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.617500 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" Oct 30 02:05:23.617531 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.617518 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf" event={"ID":"f8e03b95-821f-456c-a435-831d0bfa52f0","Type":"ContainerDied","Data":"2e7458336187837c34141481eeaa049b0ea014c570cbaf09b21b7a0818cf30e9"} Oct 30 02:05:23.618083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.617582 5998 scope.go:122] "RemoveContainer" containerID="c56a2860f48966a3356b49c3b13c68743ff64be9de71f58be01bd78e7e4ff846" Oct 30 02:05:23.642350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.642320 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead31f45-5cfc-4910-a2bd-1e4736f1268a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3054e243-d4rd6\" (UID: \"ead31f45-5cfc-4910-a2bd-1e4736f1268a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6" Oct 30 02:05:23.642524 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.642472 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead31f45-5cfc-4910-a2bd-1e4736f1268a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3054e243-d4rd6\" (UID: \"ead31f45-5cfc-4910-a2bd-1e4736f1268a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6" Oct 30 02:05:23.642524 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.642515 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx9g4\" (UniqueName: \"kubernetes.io/projected/ead31f45-5cfc-4910-a2bd-1e4736f1268a-kube-api-access-lx9g4\") pod \"maas-api-revoke-sub-3054e243-d4rd6\" (UID: \"ead31f45-5cfc-4910-a2bd-1e4736f1268a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6" Oct 30 02:05:23.665714 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.665678 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf"] Oct 30 02:05:23.671255 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.671226 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5c90f5-openshift-default-6876c6b675-lbxpf"] Oct 30 02:05:23.743701 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.743596 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead31f45-5cfc-4910-a2bd-1e4736f1268a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3054e243-d4rd6\" (UID: \"ead31f45-5cfc-4910-a2bd-1e4736f1268a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6" Oct 30 02:05:23.743701 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.743669 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead31f45-5cfc-4910-a2bd-1e4736f1268a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3054e243-d4rd6\" (UID: \"ead31f45-5cfc-4910-a2bd-1e4736f1268a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6" Oct 30 02:05:23.743701 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.743703 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx9g4\" (UniqueName: \"kubernetes.io/projected/ead31f45-5cfc-4910-a2bd-1e4736f1268a-kube-api-access-lx9g4\") pod \"maas-api-revoke-sub-3054e243-d4rd6\" (UID: \"ead31f45-5cfc-4910-a2bd-1e4736f1268a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6" Oct 30 02:05:23.744472 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.744446 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead31f45-5cfc-4910-a2bd-1e4736f1268a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3054e243-d4rd6\" (UID: \"ead31f45-5cfc-4910-a2bd-1e4736f1268a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6" Oct 30 02:05:23.746122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.746103 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead31f45-5cfc-4910-a2bd-1e4736f1268a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3054e243-d4rd6\" (UID: \"ead31f45-5cfc-4910-a2bd-1e4736f1268a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6" Oct 30 02:05:23.765822 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.765790 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx9g4\" (UniqueName: \"kubernetes.io/projected/ead31f45-5cfc-4910-a2bd-1e4736f1268a-kube-api-access-lx9g4\") pod \"maas-api-revoke-sub-3054e243-d4rd6\" (UID: \"ead31f45-5cfc-4910-a2bd-1e4736f1268a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6" Oct 30 02:05:23.886178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:23.886146 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6" Oct 30 02:05:24.197300 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:24.197266 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6"] Oct 30 02:05:24.201328 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:05:24.201300 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead31f45_5cfc_4910_a2bd_1e4736f1268a.slice/crio-96fb4facacda5e2cb9295fbc0b74cc7f077ffd9bf4ad4131b462e117317224be WatchSource:0}: Error finding container 96fb4facacda5e2cb9295fbc0b74cc7f077ffd9bf4ad4131b462e117317224be: Status 404 returned error can't find the container with id 96fb4facacda5e2cb9295fbc0b74cc7f077ffd9bf4ad4131b462e117317224be Oct 30 02:05:24.540745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:24.540706 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8e03b95-821f-456c-a435-831d0bfa52f0" path="/var/lib/kubelet/pods/f8e03b95-821f-456c-a435-831d0bfa52f0/volumes" Oct 30 02:05:24.628738 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:24.628690 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6" event={"ID":"ead31f45-5cfc-4910-a2bd-1e4736f1268a","Type":"ContainerStarted","Data":"e5813b1a1c0b84e143f61a4721a8ba8dfa08787fa2cd8649e22fcf2a3633dabc"} Oct 30 02:05:24.629150 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:24.628759 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6" event={"ID":"ead31f45-5cfc-4910-a2bd-1e4736f1268a","Type":"ContainerStarted","Data":"96fb4facacda5e2cb9295fbc0b74cc7f077ffd9bf4ad4131b462e117317224be"} Oct 30 02:05:24.669536 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:24.669479 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6" podStartSLOduration=1.6694628809999998 podStartE2EDuration="1.669462881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:24.666592132 +0000 UTC m=+2154.245383215" watchObservedRunningTime="2026-10-30 02:05:24.669462881 +0000 UTC m=+2154.248253961" Oct 30 02:05:25.182569 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:25.182535 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:25.216965 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:05:25.216925 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:05:25.288901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:25.288866 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:25.638449 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:25.638398 5998 generic.go:356] "Generic (PLEG): container finished" podID="ead31f45-5cfc-4910-a2bd-1e4736f1268a" containerID="e5813b1a1c0b84e143f61a4721a8ba8dfa08787fa2cd8649e22fcf2a3633dabc" exitCode=0 Oct 30 02:05:25.638961 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:25.638464 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6" event={"ID":"ead31f45-5cfc-4910-a2bd-1e4736f1268a","Type":"ContainerDied","Data":"e5813b1a1c0b84e143f61a4721a8ba8dfa08787fa2cd8649e22fcf2a3633dabc"} Oct 30 02:05:26.812413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:26.812386 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6" Oct 30 02:05:26.885503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:26.885470 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ead31f45-5cfc-4910-a2bd-1e4736f1268a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead31f45-5cfc-4910-a2bd-1e4736f1268a-maas-api-xks-ca\") pod \"ead31f45-5cfc-4910-a2bd-1e4736f1268a\" (UID: \"ead31f45-5cfc-4910-a2bd-1e4736f1268a\") " Oct 30 02:05:26.885503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:26.885514 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead31f45-5cfc-4910-a2bd-1e4736f1268a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead31f45-5cfc-4910-a2bd-1e4736f1268a-maas-api-openshift-ca\") pod \"ead31f45-5cfc-4910-a2bd-1e4736f1268a\" (UID: \"ead31f45-5cfc-4910-a2bd-1e4736f1268a\") " Oct 30 02:05:26.885856 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:26.885580 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead31f45-5cfc-4910-a2bd-1e4736f1268a-kube-api-access-lx9g4\" (UniqueName: \"kubernetes.io/projected/ead31f45-5cfc-4910-a2bd-1e4736f1268a-kube-api-access-lx9g4\") pod \"ead31f45-5cfc-4910-a2bd-1e4736f1268a\" (UID: \"ead31f45-5cfc-4910-a2bd-1e4736f1268a\") " Oct 30 02:05:26.886414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:26.886378 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead31f45-5cfc-4910-a2bd-1e4736f1268a-maas-api-openshift-ca" pod "ead31f45-5cfc-4910-a2bd-1e4736f1268a" (UID: "ead31f45-5cfc-4910-a2bd-1e4736f1268a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:26.888370 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:26.888342 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead31f45-5cfc-4910-a2bd-1e4736f1268a-kube-api-access-lx9g4" pod "ead31f45-5cfc-4910-a2bd-1e4736f1268a" (UID: "ead31f45-5cfc-4910-a2bd-1e4736f1268a"). InnerVolumeSpecName "kube-api-access-lx9g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:26.903377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:26.903345 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead31f45-5cfc-4910-a2bd-1e4736f1268a-maas-api-xks-ca" pod "ead31f45-5cfc-4910-a2bd-1e4736f1268a" (UID: "ead31f45-5cfc-4910-a2bd-1e4736f1268a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:26.987028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:26.986989 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead31f45-5cfc-4910-a2bd-1e4736f1268a-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:26.987028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:26.987023 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead31f45-5cfc-4910-a2bd-1e4736f1268a-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:26.987028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:26.987036 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx9g4\" (UniqueName: \"kubernetes.io/projected/ead31f45-5cfc-4910-a2bd-1e4736f1268a-kube-api-access-lx9g4\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:27.090919 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.090885 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:27.470634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.470593 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298"] Oct 30 02:05:27.471253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.471221 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ead31f45-5cfc-4910-a2bd-1e4736f1268a" containerName="revoke-keys" Oct 30 02:05:27.471253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.471253 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead31f45-5cfc-4910-a2bd-1e4736f1268a" containerName="revoke-keys" Oct 30 02:05:27.471444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.471388 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead31f45-5cfc-4910-a2bd-1e4736f1268a" containerName="revoke-keys" Oct 30 02:05:27.474702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.474683 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298" Oct 30 02:05:27.492736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.492705 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b838ef26-v9298\" (UID: \"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298" Oct 30 02:05:27.492845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.492831 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b838ef26-v9298\" (UID: \"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298" Oct 30 02:05:27.492900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.492877 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbj8w\" (UniqueName: \"kubernetes.io/projected/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-kube-api-access-gbj8w\") pod \"maas-api-revoke-sub-b838ef26-v9298\" (UID: \"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298" Oct 30 02:05:27.495847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.495823 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298"] Oct 30 02:05:27.594041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.593999 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbj8w\" (UniqueName: \"kubernetes.io/projected/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-kube-api-access-gbj8w\") pod \"maas-api-revoke-sub-b838ef26-v9298\" (UID: \"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298" Oct 30 02:05:27.594274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.594190 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b838ef26-v9298\" (UID: \"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298" Oct 30 02:05:27.594353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.594315 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b838ef26-v9298\" (UID: \"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298" Oct 30 02:05:27.594860 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.594838 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b838ef26-v9298\" (UID: \"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298" Oct 30 02:05:27.596612 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.596588 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b838ef26-v9298\" (UID: \"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298" Oct 30 02:05:27.615519 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.615489 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbj8w\" (UniqueName: \"kubernetes.io/projected/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-kube-api-access-gbj8w\") pod \"maas-api-revoke-sub-b838ef26-v9298\" (UID: \"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298" Oct 30 02:05:27.657147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.657080 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6" Oct 30 02:05:27.657147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.657117 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6" event={"ID":"ead31f45-5cfc-4910-a2bd-1e4736f1268a","Type":"ContainerDied","Data":"96fb4facacda5e2cb9295fbc0b74cc7f077ffd9bf4ad4131b462e117317224be"} Oct 30 02:05:27.657431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.657176 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96fb4facacda5e2cb9295fbc0b74cc7f077ffd9bf4ad4131b462e117317224be" Oct 30 02:05:27.787342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:27.787260 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298" Oct 30 02:05:28.107991 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:28.107967 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298"] Oct 30 02:05:28.110849 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:05:28.110819 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11889f6_1ef3_4f9e_80f0_131f9c19f2f6.slice/crio-44e6de00d3a00b3971e8f6482c10e47e293269cb824253a1644406b0a1e47360 WatchSource:0}: Error finding container 44e6de00d3a00b3971e8f6482c10e47e293269cb824253a1644406b0a1e47360: Status 404 returned error can't find the container with id 44e6de00d3a00b3971e8f6482c10e47e293269cb824253a1644406b0a1e47360 Oct 30 02:05:28.672986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:28.672884 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298" event={"ID":"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6","Type":"ContainerStarted","Data":"7812ef933f3b5ce6ed5828e49344d651c7cf9543dd9fd59228ac17bb385e41f1"} Oct 30 02:05:28.673234 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:28.672985 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298" event={"ID":"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6","Type":"ContainerStarted","Data":"44e6de00d3a00b3971e8f6482c10e47e293269cb824253a1644406b0a1e47360"} Oct 30 02:05:28.713212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:28.713157 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298" podStartSLOduration=1.713143219 podStartE2EDuration="1.713143219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:28.709550611 +0000 UTC m=+2158.288341691" watchObservedRunningTime="2026-10-30 02:05:28.713143219 +0000 UTC m=+2158.291934297" Oct 30 02:05:28.895354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:28.895320 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:29.680518 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:29.680484 5998 generic.go:356] "Generic (PLEG): container finished" podID="e11889f6-1ef3-4f9e-80f0-131f9c19f2f6" containerID="7812ef933f3b5ce6ed5828e49344d651c7cf9543dd9fd59228ac17bb385e41f1" exitCode=0 Oct 30 02:05:29.680931 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:29.680566 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298" event={"ID":"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6","Type":"ContainerDied","Data":"7812ef933f3b5ce6ed5828e49344d651c7cf9543dd9fd59228ac17bb385e41f1"} Oct 30 02:05:30.846557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.846529 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298" Oct 30 02:05:30.884899 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.884867 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph"] Oct 30 02:05:30.885421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.885406 5998 cpu_manager.go:414] "containerMap: removing container" podUID="e11889f6-1ef3-4f9e-80f0-131f9c19f2f6" containerName="revoke-keys" Oct 30 02:05:30.885421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.885422 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e11889f6-1ef3-4f9e-80f0-131f9c19f2f6" containerName="revoke-keys" Oct 30 02:05:30.885533 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.885505 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="e11889f6-1ef3-4f9e-80f0-131f9c19f2f6" containerName="revoke-keys" Oct 30 02:05:30.888755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.888733 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:30.901708 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.901679 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k"] Oct 30 02:05:30.906132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.903927 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b93d51-openshift-default-dockercfg-qh2lt\"" Oct 30 02:05:30.906273 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.906186 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:30.922475 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.922217 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph"] Oct 30 02:05:30.942901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.942808 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87"] Oct 30 02:05:30.944646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.944592 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-kube-api-access-gbj8w\" (UniqueName: \"kubernetes.io/projected/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-kube-api-access-gbj8w\") pod \"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6\" (UID: \"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6\") " Oct 30 02:05:30.944857 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.944835 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-maas-api-xks-ca\") pod \"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6\" (UID: \"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6\") " Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.944934 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-maas-api-openshift-ca\") pod \"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6\" (UID: \"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6\") " Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.945204 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389bcbb5-189b-45d6-8154-e542936cf39b-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.945239 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.945271 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbnqg\" (UniqueName: \"kubernetes.io/projected/389bcbb5-189b-45d6-8154-e542936cf39b-kube-api-access-pbnqg\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.945316 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.945351 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.945375 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389bcbb5-189b-45d6-8154-e542936cf39b-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.945419 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389bcbb5-189b-45d6-8154-e542936cf39b-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.945464 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.945513 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.945548 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.945586 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.945618 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.945882 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.945974 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.946044 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sbhc\" (UniqueName: \"kubernetes.io/projected/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-kube-api-access-6sbhc\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.946122 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.946158 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:30.946389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.946190 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:30.947855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.947823 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-maas-api-openshift-ca" pod "e11889f6-1ef3-4f9e-80f0-131f9c19f2f6" (UID: "e11889f6-1ef3-4f9e-80f0-131f9c19f2f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:30.949835 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.949808 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:30.956137 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.953394 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-maas-api-xks-ca" pod "e11889f6-1ef3-4f9e-80f0-131f9c19f2f6" (UID: "e11889f6-1ef3-4f9e-80f0-131f9c19f2f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:30.956752 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.956724 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-kube-api-access-gbj8w" pod "e11889f6-1ef3-4f9e-80f0-131f9c19f2f6" (UID: "e11889f6-1ef3-4f9e-80f0-131f9c19f2f6"). InnerVolumeSpecName "kube-api-access-gbj8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:30.960644 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.960562 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k"] Oct 30 02:05:30.974130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:30.973499 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87"] Oct 30 02:05:31.047270 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.047229 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rjwb\" (UniqueName: \"kubernetes.io/projected/c13ee6d9-ce09-49b9-830f-5b8b25506581-kube-api-access-5rjwb\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.047453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.047277 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389bcbb5-189b-45d6-8154-e542936cf39b-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.047453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.047296 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.047453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.047320 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbnqg\" (UniqueName: \"kubernetes.io/projected/389bcbb5-189b-45d6-8154-e542936cf39b-kube-api-access-pbnqg\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.047453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.047389 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.047672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.047457 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.047672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.047491 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.047672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.047574 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.047672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.047615 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389bcbb5-189b-45d6-8154-e542936cf39b-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.047672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.047661 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.047955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.047786 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389bcbb5-189b-45d6-8154-e542936cf39b-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.047955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.047834 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.047955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.047869 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.047955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.047915 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.047955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.047949 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.048332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.047962 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.048332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048011 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.048332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048187 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.048332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048210 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.048332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048271 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.048332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048321 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.048652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048354 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.048652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048368 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.048652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048447 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.048652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048528 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13ee6d9-ce09-49b9-830f-5b8b25506581-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.048652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048560 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.048652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048593 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sbhc\" (UniqueName: \"kubernetes.io/projected/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-kube-api-access-6sbhc\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.048652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048607 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.048652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048629 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.049102 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048705 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.049102 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048752 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.049102 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048789 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.049102 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048818 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.049102 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048872 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389bcbb5-189b-45d6-8154-e542936cf39b-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.049102 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.048885 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.049102 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.049080 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.049500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.049113 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:31.049500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.049139 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:31.049500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.049167 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.049500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.049175 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbj8w\" (UniqueName: \"kubernetes.io/projected/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6-kube-api-access-gbj8w\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:31.050723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.050698 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389bcbb5-189b-45d6-8154-e542936cf39b-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.050931 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.050911 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.051362 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.051338 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.051362 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.051359 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.069813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.069746 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbnqg\" (UniqueName: \"kubernetes.io/projected/389bcbb5-189b-45d6-8154-e542936cf39b-kube-api-access-pbnqg\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.070083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.070061 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389bcbb5-189b-45d6-8154-e542936cf39b-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.070083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.070071 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.070252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.070071 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sbhc\" (UniqueName: \"kubernetes.io/projected/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-kube-api-access-6sbhc\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.150400 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.150359 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.150588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.150472 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.150588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.150537 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rjwb\" (UniqueName: \"kubernetes.io/projected/c13ee6d9-ce09-49b9-830f-5b8b25506581-kube-api-access-5rjwb\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.150588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.150578 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.150770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.150628 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.150770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.150675 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.150770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.150727 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.150926 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.150820 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.150926 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.150877 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-credential-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.150926 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.150901 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13ee6d9-ce09-49b9-830f-5b8b25506581-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.151116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.151010 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-workload-certs\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.151116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.151059 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-workload-socket\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.151246 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.151135 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-data\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.151577 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.151553 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13ee6d9-ce09-49b9-830f-5b8b25506581-istiod-ca-cert\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.152772 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.152754 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-envoy\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.153158 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.153141 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-podinfo\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.171630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.171595 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-token\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.171758 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.171674 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rjwb\" (UniqueName: \"kubernetes.io/projected/c13ee6d9-ce09-49b9-830f-5b8b25506581-kube-api-access-5rjwb\") pod \"e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.208480 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.208391 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:31.223198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.223174 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:31.297021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.296766 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:31.549160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.548800 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph"] Oct 30 02:05:31.552464 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:05:31.552435 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0e07e1_c219_46aa_80e2_bf0eaf2ec85e.slice/crio-564c523a964f8d4d8f373e54be2b60badbf3efe46b096c94b468cc325e70a3e3 WatchSource:0}: Error finding container 564c523a964f8d4d8f373e54be2b60badbf3efe46b096c94b468cc325e70a3e3: Status 404 returned error can't find the container with id 564c523a964f8d4d8f373e54be2b60badbf3efe46b096c94b468cc325e70a3e3 Oct 30 02:05:31.552843 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:05:31.552822 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389bcbb5_189b_45d6_8154_e542936cf39b.slice/crio-cfbc5e62f2d555a04072f42722d5088877da0b7485853d6e655408f14da984c0 WatchSource:0}: Error finding container cfbc5e62f2d555a04072f42722d5088877da0b7485853d6e655408f14da984c0: Status 404 returned error can't find the container with id cfbc5e62f2d555a04072f42722d5088877da0b7485853d6e655408f14da984c0 Oct 30 02:05:31.553383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.553358 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k"] Oct 30 02:05:31.555429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.555400 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:05:31.555549 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.555460 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:05:31.555549 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.555488 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:05:31.555662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.555630 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:05:31.555722 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.555686 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:05:31.555722 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.555717 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:05:31.626411 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.625289 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87"] Oct 30 02:05:31.629648 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:05:31.629621 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13ee6d9_ce09_49b9_830f_5b8b25506581.slice/crio-dcf44a2e0356db513113193fccc341a72e009c878c2ca9f4400bf163c35ea11c WatchSource:0}: Error finding container dcf44a2e0356db513113193fccc341a72e009c878c2ca9f4400bf163c35ea11c: Status 404 returned error can't find the container with id dcf44a2e0356db513113193fccc341a72e009c878c2ca9f4400bf163c35ea11c Oct 30 02:05:31.633365 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.633326 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:05:31.633474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.633419 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:05:31.633474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.633463 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:05:31.722047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.717078 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:31.722047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.721769 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" event={"ID":"c13ee6d9-ce09-49b9-830f-5b8b25506581","Type":"ContainerStarted","Data":"dcf44a2e0356db513113193fccc341a72e009c878c2ca9f4400bf163c35ea11c"} Oct 30 02:05:31.723183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.723157 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298" event={"ID":"e11889f6-1ef3-4f9e-80f0-131f9c19f2f6","Type":"ContainerDied","Data":"44e6de00d3a00b3971e8f6482c10e47e293269cb824253a1644406b0a1e47360"} Oct 30 02:05:31.723353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.723342 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44e6de00d3a00b3971e8f6482c10e47e293269cb824253a1644406b0a1e47360" Oct 30 02:05:31.723528 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.723517 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298" Oct 30 02:05:31.731297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.730214 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" event={"ID":"389bcbb5-189b-45d6-8154-e542936cf39b","Type":"ContainerStarted","Data":"c3edf767af4b3e3bd17eb839f79426cfc9ee1609559eda0bf9293052c0d3c21d"} Oct 30 02:05:31.731297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.730270 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" event={"ID":"389bcbb5-189b-45d6-8154-e542936cf39b","Type":"ContainerStarted","Data":"cfbc5e62f2d555a04072f42722d5088877da0b7485853d6e655408f14da984c0"} Oct 30 02:05:31.738303 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.737444 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" event={"ID":"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e","Type":"ContainerStarted","Data":"cb0b55c49f2ada5676d0d7b8de4dcc9cf8ff84c1ac3da34f992344ce5e0cf39b"} Oct 30 02:05:31.738303 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.737498 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" event={"ID":"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e","Type":"ContainerStarted","Data":"564c523a964f8d4d8f373e54be2b60badbf3efe46b096c94b468cc325e70a3e3"} Oct 30 02:05:31.777147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:31.776183 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" podStartSLOduration=1.776137557 podStartE2EDuration="1.776137557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:31.773797783 +0000 UTC m=+2161.352588865" watchObservedRunningTime="2026-10-30 02:05:31.776137557 +0000 UTC m=+2161.354928638" Oct 30 02:05:32.209338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:32.209297 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:32.216594 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:32.216231 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 02:05:32.216594 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:32.216294 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" podUID="1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 02:05:32.226851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:32.226777 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:32.247139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:32.244827 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 02:05:32.247139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:32.244913 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" podUID="389bcbb5-189b-45d6-8154-e542936cf39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 02:05:32.784116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:32.781894 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" event={"ID":"c13ee6d9-ce09-49b9-830f-5b8b25506581","Type":"ContainerStarted","Data":"4883db70d3ca9e4126e230d48adbf4f534998dc62c149f7f27bc8be7c0d4ef6b"} Oct 30 02:05:32.841127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:32.840884 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" podStartSLOduration=2.840865356 podStartE2EDuration="2.840865356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:31.855263429 +0000 UTC m=+2161.434054511" watchObservedRunningTime="2026-10-30 02:05:32.840865356 +0000 UTC m=+2162.419656438" Oct 30 02:05:32.867113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:32.863696 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" podStartSLOduration=2.863673603 podStartE2EDuration="2.863673603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:32.839278728 +0000 UTC m=+2162.418069822" watchObservedRunningTime="2026-10-30 02:05:32.863673603 +0000 UTC m=+2162.442464684" Oct 30 02:05:33.216709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:33.215779 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 02:05:33.216709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:33.215840 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" podUID="1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 02:05:33.223722 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:33.223583 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 02:05:33.223722 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:33.223643 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" podUID="389bcbb5-189b-45d6-8154-e542936cf39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 02:05:33.313141 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:33.305311 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:33.334115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:33.318652 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 02:05:33.334115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:33.318756 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" podUID="c13ee6d9-ce09-49b9-830f-5b8b25506581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 02:05:34.216249 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:34.216209 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 02:05:34.216974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:34.216939 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" podUID="1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 02:05:34.230523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:34.230489 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 02:05:34.230817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:34.230786 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" podUID="389bcbb5-189b-45d6-8154-e542936cf39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 02:05:34.303127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:34.301199 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 02:05:34.303127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:34.301273 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" podUID="c13ee6d9-ce09-49b9-830f-5b8b25506581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 02:05:34.692116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:34.689154 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:35.218116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:35.212717 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 02:05:35.218116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:35.212793 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" podUID="1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 02:05:35.227123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:35.226458 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 02:05:35.227123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:35.226522 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" podUID="389bcbb5-189b-45d6-8154-e542936cf39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 02:05:35.299659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:35.299505 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 02:05:35.299659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:35.299565 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" podUID="c13ee6d9-ce09-49b9-830f-5b8b25506581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 02:05:36.121116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:36.110530 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:36.221547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:36.221502 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:36.222047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:36.222015 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 02:05:36.222135 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:36.222070 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" podUID="1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 02:05:36.226116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:36.224047 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 02:05:36.226116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:36.224121 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" podUID="389bcbb5-189b-45d6-8154-e542936cf39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 02:05:36.299952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:36.299916 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 02:05:36.300167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:36.299974 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" podUID="c13ee6d9-ce09-49b9-830f-5b8b25506581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 02:05:36.494356 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:36.492861 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:36.681964 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:36.679504 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs"] Oct 30 02:05:36.693444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:36.693404 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs" podUID="eb3049cb-dd1f-484c-addf-a9cf4285da41" containerName="payload-pre-processing" containerID="cri-o://7e7806268237253627fe8ef6a16a5118853f30246c177f80e0551f0750266286" gracePeriod=30 Oct 30 02:05:36.748297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:36.723068 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:36.859183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:36.840267 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:37.222146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:37.222107 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 02:05:37.222833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:37.222722 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" podUID="1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 02:05:37.224162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:37.224139 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 02:05:37.224329 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:37.224305 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" podUID="389bcbb5-189b-45d6-8154-e542936cf39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 02:05:37.301639 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:37.301522 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 02:05:37.301639 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:37.301581 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" podUID="c13ee6d9-ce09-49b9-830f-5b8b25506581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 02:05:37.619251 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:37.603677 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:37.737128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:37.733853 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:37.818578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:37.818528 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:37.935255 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:37.935205 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:38.098577 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.067488 5998 generic.go:356] "Generic (PLEG): container finished" podID="eb3049cb-dd1f-484c-addf-a9cf4285da41" containerID="7e7806268237253627fe8ef6a16a5118853f30246c177f80e0551f0750266286" exitCode=0 Oct 30 02:05:38.098577 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.067626 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs" event={"ID":"eb3049cb-dd1f-484c-addf-a9cf4285da41","Type":"ContainerDied","Data":"7e7806268237253627fe8ef6a16a5118853f30246c177f80e0551f0750266286"} Oct 30 02:05:38.245216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.221053 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 02:05:38.245216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.221141 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" podUID="1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 02:05:38.245216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.227405 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs" Oct 30 02:05:38.245216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.229661 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 02:05:38.245216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.229711 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" podUID="389bcbb5-189b-45d6-8154-e542936cf39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 02:05:38.327534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.301273 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 02:05:38.327534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.301377 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" podUID="c13ee6d9-ce09-49b9-830f-5b8b25506581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 02:05:38.383223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.382982 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb3049cb-dd1f-484c-addf-a9cf4285da41-kube-api-access-cf2nm\" (UniqueName: \"kubernetes.io/projected/eb3049cb-dd1f-484c-addf-a9cf4285da41-kube-api-access-cf2nm\") pod \"eb3049cb-dd1f-484c-addf-a9cf4285da41\" (UID: \"eb3049cb-dd1f-484c-addf-a9cf4285da41\") " Oct 30 02:05:38.383421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.383275 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb3049cb-dd1f-484c-addf-a9cf4285da41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3049cb-dd1f-484c-addf-a9cf4285da41-plugins-config-volume\") pod \"eb3049cb-dd1f-484c-addf-a9cf4285da41\" (UID: \"eb3049cb-dd1f-484c-addf-a9cf4285da41\") " Oct 30 02:05:38.384892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.384863 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb3049cb-dd1f-484c-addf-a9cf4285da41-plugins-config-volume" pod "eb3049cb-dd1f-484c-addf-a9cf4285da41" (UID: "eb3049cb-dd1f-484c-addf-a9cf4285da41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:38.388343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.388166 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3049cb-dd1f-484c-addf-a9cf4285da41-kube-api-access-cf2nm" pod "eb3049cb-dd1f-484c-addf-a9cf4285da41" (UID: "eb3049cb-dd1f-484c-addf-a9cf4285da41"). InnerVolumeSpecName "kube-api-access-cf2nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:38.485373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.485328 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf2nm\" (UniqueName: \"kubernetes.io/projected/eb3049cb-dd1f-484c-addf-a9cf4285da41-kube-api-access-cf2nm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:38.485373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.485374 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb3049cb-dd1f-484c-addf-a9cf4285da41-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:38.512671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.512576 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t"] Oct 30 02:05:38.513432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.513408 5998 cpu_manager.go:414] "containerMap: removing container" podUID="eb3049cb-dd1f-484c-addf-a9cf4285da41" containerName="payload-pre-processing" Oct 30 02:05:38.513511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.513436 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb3049cb-dd1f-484c-addf-a9cf4285da41" containerName="payload-pre-processing" Oct 30 02:05:38.513656 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.513641 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb3049cb-dd1f-484c-addf-a9cf4285da41" containerName="payload-pre-processing" Oct 30 02:05:38.517766 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.517738 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t" Oct 30 02:05:38.535976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.535948 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:05:38.536809 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.536786 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:05:38.536941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.536929 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:05:38.544244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.544121 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:05:38.553386 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.553358 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t"] Oct 30 02:05:38.688413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.688367 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24x6v\" (UniqueName: \"kubernetes.io/projected/65158b49-402f-48d8-8335-e6cb600e299f-kube-api-access-24x6v\") pod \"maas-api-revoke-sub-2ff95b39-rf42t\" (UID: \"65158b49-402f-48d8-8335-e6cb600e299f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t" Oct 30 02:05:38.688628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.688478 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65158b49-402f-48d8-8335-e6cb600e299f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ff95b39-rf42t\" (UID: \"65158b49-402f-48d8-8335-e6cb600e299f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t" Oct 30 02:05:38.688628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.688590 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65158b49-402f-48d8-8335-e6cb600e299f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ff95b39-rf42t\" (UID: \"65158b49-402f-48d8-8335-e6cb600e299f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t" Oct 30 02:05:38.790692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.790598 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65158b49-402f-48d8-8335-e6cb600e299f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ff95b39-rf42t\" (UID: \"65158b49-402f-48d8-8335-e6cb600e299f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t" Oct 30 02:05:38.790872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.790804 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24x6v\" (UniqueName: \"kubernetes.io/projected/65158b49-402f-48d8-8335-e6cb600e299f-kube-api-access-24x6v\") pod \"maas-api-revoke-sub-2ff95b39-rf42t\" (UID: \"65158b49-402f-48d8-8335-e6cb600e299f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t" Oct 30 02:05:38.790946 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.790897 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65158b49-402f-48d8-8335-e6cb600e299f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ff95b39-rf42t\" (UID: \"65158b49-402f-48d8-8335-e6cb600e299f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t" Oct 30 02:05:38.791755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.791721 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65158b49-402f-48d8-8335-e6cb600e299f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ff95b39-rf42t\" (UID: \"65158b49-402f-48d8-8335-e6cb600e299f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t" Oct 30 02:05:38.793066 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.793040 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65158b49-402f-48d8-8335-e6cb600e299f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ff95b39-rf42t\" (UID: \"65158b49-402f-48d8-8335-e6cb600e299f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t" Oct 30 02:05:38.812204 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.812176 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24x6v\" (UniqueName: \"kubernetes.io/projected/65158b49-402f-48d8-8335-e6cb600e299f-kube-api-access-24x6v\") pod \"maas-api-revoke-sub-2ff95b39-rf42t\" (UID: \"65158b49-402f-48d8-8335-e6cb600e299f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t" Oct 30 02:05:38.849853 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:38.849818 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t" Oct 30 02:05:39.079516 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:39.079423 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs" event={"ID":"eb3049cb-dd1f-484c-addf-a9cf4285da41","Type":"ContainerDied","Data":"4832267aa7ca2b2b91e3aa7f5fa8fc4fdc07c586d9a9492ba3c83becff0dddd3"} Oct 30 02:05:39.079516 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:39.079510 5998 scope.go:122] "RemoveContainer" containerID="7e7806268237253627fe8ef6a16a5118853f30246c177f80e0551f0750266286" Oct 30 02:05:39.079737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:39.079686 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs" Oct 30 02:05:39.148514 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:39.148482 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs"] Oct 30 02:05:39.152191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:39.152161 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-168989-55fbb7b799-n85cs"] Oct 30 02:05:39.201568 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:39.201535 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t"] Oct 30 02:05:39.232351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:39.231194 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 02:05:39.232351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:39.231254 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" podUID="1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 02:05:39.232351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:39.231904 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 02:05:39.232351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:39.231943 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" podUID="389bcbb5-189b-45d6-8154-e542936cf39b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 02:05:39.569931 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:39.569895 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:39.875109 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:39.875043 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:39.876641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:39.876607 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:39.880392 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:39.880334 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:05:40.095143 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:40.095042 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t" event={"ID":"65158b49-402f-48d8-8335-e6cb600e299f","Type":"ContainerStarted","Data":"d51fc1f8096f35f11ea343e4ae01260202c2879fb8f054a73ef0984db8268e6d"} Oct 30 02:05:40.095143 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:40.095136 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t" event={"ID":"65158b49-402f-48d8-8335-e6cb600e299f","Type":"ContainerStarted","Data":"ae646a6ed9597aa08e8809dec853e8195dc799a4c65866f56f046a163a5f51a8"} Oct 30 02:05:40.134756 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:40.134678 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t" podStartSLOduration=2.134664362 podStartE2EDuration="2.134664362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:40.132294779 +0000 UTC m=+2169.711085862" watchObservedRunningTime="2026-10-30 02:05:40.134664362 +0000 UTC m=+2169.713455443" Oct 30 02:05:40.213747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:40.213714 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:40.214142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:40.214117 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:40.214848 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:40.214825 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:05:40.228842 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:40.228817 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:40.229169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:40.229143 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:40.229826 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:40.229800 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:05:40.538970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:40.538926 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb3049cb-dd1f-484c-addf-a9cf4285da41" path="/var/lib/kubelet/pods/eb3049cb-dd1f-484c-addf-a9cf4285da41/volumes" Oct 30 02:05:41.105925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:41.105890 5998 generic.go:356] "Generic (PLEG): container finished" podID="65158b49-402f-48d8-8335-e6cb600e299f" containerID="d51fc1f8096f35f11ea343e4ae01260202c2879fb8f054a73ef0984db8268e6d" exitCode=0 Oct 30 02:05:41.106454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:41.105924 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t" event={"ID":"65158b49-402f-48d8-8335-e6cb600e299f","Type":"ContainerDied","Data":"d51fc1f8096f35f11ea343e4ae01260202c2879fb8f054a73ef0984db8268e6d"} Oct 30 02:05:41.280908 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:41.280874 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:42.279491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:42.279464 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t" Oct 30 02:05:42.383461 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:42.383382 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65158b49-402f-48d8-8335-e6cb600e299f-kube-api-access-24x6v\" (UniqueName: \"kubernetes.io/projected/65158b49-402f-48d8-8335-e6cb600e299f-kube-api-access-24x6v\") pod \"65158b49-402f-48d8-8335-e6cb600e299f\" (UID: \"65158b49-402f-48d8-8335-e6cb600e299f\") " Oct 30 02:05:42.383624 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:42.383488 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65158b49-402f-48d8-8335-e6cb600e299f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65158b49-402f-48d8-8335-e6cb600e299f-maas-api-xks-ca\") pod \"65158b49-402f-48d8-8335-e6cb600e299f\" (UID: \"65158b49-402f-48d8-8335-e6cb600e299f\") " Oct 30 02:05:42.383624 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:42.383570 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65158b49-402f-48d8-8335-e6cb600e299f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65158b49-402f-48d8-8335-e6cb600e299f-maas-api-openshift-ca\") pod \"65158b49-402f-48d8-8335-e6cb600e299f\" (UID: \"65158b49-402f-48d8-8335-e6cb600e299f\") " Oct 30 02:05:42.383924 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:42.383896 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65158b49-402f-48d8-8335-e6cb600e299f-maas-api-openshift-ca" pod "65158b49-402f-48d8-8335-e6cb600e299f" (UID: "65158b49-402f-48d8-8335-e6cb600e299f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:42.384065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:42.383998 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65158b49-402f-48d8-8335-e6cb600e299f-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:42.385589 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:42.385562 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65158b49-402f-48d8-8335-e6cb600e299f-maas-api-xks-ca" pod "65158b49-402f-48d8-8335-e6cb600e299f" (UID: "65158b49-402f-48d8-8335-e6cb600e299f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:42.385729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:42.385709 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65158b49-402f-48d8-8335-e6cb600e299f-kube-api-access-24x6v" pod "65158b49-402f-48d8-8335-e6cb600e299f" (UID: "65158b49-402f-48d8-8335-e6cb600e299f"). InnerVolumeSpecName "kube-api-access-24x6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:42.485596 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:42.485559 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24x6v\" (UniqueName: \"kubernetes.io/projected/65158b49-402f-48d8-8335-e6cb600e299f-kube-api-access-24x6v\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:42.485596 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:42.485593 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65158b49-402f-48d8-8335-e6cb600e299f-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:42.624447 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:42.624403 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:42.780586 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:42.780505 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:42.887442 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:42.887394 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:43.163080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.163034 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t" event={"ID":"65158b49-402f-48d8-8335-e6cb600e299f","Type":"ContainerDied","Data":"ae646a6ed9597aa08e8809dec853e8195dc799a4c65866f56f046a163a5f51a8"} Oct 30 02:05:43.163244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.163125 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae646a6ed9597aa08e8809dec853e8195dc799a4c65866f56f046a163a5f51a8" Oct 30 02:05:43.163244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.163204 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t" Oct 30 02:05:43.217890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.217854 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn"] Oct 30 02:05:43.218770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.218747 5998 cpu_manager.go:414] "containerMap: removing container" podUID="65158b49-402f-48d8-8335-e6cb600e299f" containerName="revoke-keys" Oct 30 02:05:43.218871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.218773 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65158b49-402f-48d8-8335-e6cb600e299f" containerName="revoke-keys" Oct 30 02:05:43.218935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.218918 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="65158b49-402f-48d8-8335-e6cb600e299f" containerName="revoke-keys" Oct 30 02:05:43.237785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.237700 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.249176 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.249041 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn"] Oct 30 02:05:43.297215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.297177 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b27c952-8035-4847-b243-4471250d3d09-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.297703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.297235 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.297703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.297273 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.297703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.297305 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b27c952-8035-4847-b243-4471250d3d09-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.297703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.297403 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.297703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.297489 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b27c952-8035-4847-b243-4471250d3d09-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.297703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.297608 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn76d\" (UniqueName: \"kubernetes.io/projected/2b27c952-8035-4847-b243-4471250d3d09-kube-api-access-xn76d\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.297703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.297669 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.298009 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.297759 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.399208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.399173 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.399385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.399224 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b27c952-8035-4847-b243-4471250d3d09-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.399385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.399252 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn76d\" (UniqueName: \"kubernetes.io/projected/2b27c952-8035-4847-b243-4471250d3d09-kube-api-access-xn76d\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.399385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.399296 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.399385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.399370 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.399620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.399443 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b27c952-8035-4847-b243-4471250d3d09-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.399620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.399494 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.399620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.399538 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.399620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.399573 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b27c952-8035-4847-b243-4471250d3d09-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.399620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.399608 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-istio-data\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.399889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.399662 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-credential-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.399889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.399842 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-workload-certs\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.399889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.399843 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-workload-socket\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.400045 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.400034 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b27c952-8035-4847-b243-4471250d3d09-istiod-ca-cert\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.401891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.401861 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-istio-envoy\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.402200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.402151 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b27c952-8035-4847-b243-4471250d3d09-istio-podinfo\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.423305 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.423235 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn76d\" (UniqueName: \"kubernetes.io/projected/2b27c952-8035-4847-b243-4471250d3d09-kube-api-access-xn76d\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.423441 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.423301 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b27c952-8035-4847-b243-4471250d3d09-istio-token\") pod \"e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.473026 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.472949 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:43.591284 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.591238 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:43.715392 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.715299 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k"] Oct 30 02:05:43.721999 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.721967 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k" Oct 30 02:05:43.737042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.736616 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:05:43.737042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.736684 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:05:43.737042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.736858 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:05:43.737042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.736951 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:05:43.740863 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.740805 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k"] Oct 30 02:05:43.805867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.805825 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s27j\" (UniqueName: \"kubernetes.io/projected/9bd1e229-67e3-4acd-8c95-e821a8812050-kube-api-access-9s27j\") pod \"maas-api-revoke-sub-81202a85-rjm6k\" (UID: \"9bd1e229-67e3-4acd-8c95-e821a8812050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k" Oct 30 02:05:43.806029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.805995 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1e229-67e3-4acd-8c95-e821a8812050-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81202a85-rjm6k\" (UID: \"9bd1e229-67e3-4acd-8c95-e821a8812050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k" Oct 30 02:05:43.806122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.806056 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd1e229-67e3-4acd-8c95-e821a8812050-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81202a85-rjm6k\" (UID: \"9bd1e229-67e3-4acd-8c95-e821a8812050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k" Oct 30 02:05:43.907440 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.907402 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1e229-67e3-4acd-8c95-e821a8812050-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81202a85-rjm6k\" (UID: \"9bd1e229-67e3-4acd-8c95-e821a8812050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k" Oct 30 02:05:43.907683 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.907486 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd1e229-67e3-4acd-8c95-e821a8812050-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81202a85-rjm6k\" (UID: \"9bd1e229-67e3-4acd-8c95-e821a8812050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k" Oct 30 02:05:43.907683 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.907667 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s27j\" (UniqueName: \"kubernetes.io/projected/9bd1e229-67e3-4acd-8c95-e821a8812050-kube-api-access-9s27j\") pod \"maas-api-revoke-sub-81202a85-rjm6k\" (UID: \"9bd1e229-67e3-4acd-8c95-e821a8812050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k" Oct 30 02:05:43.908617 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.908588 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1e229-67e3-4acd-8c95-e821a8812050-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81202a85-rjm6k\" (UID: \"9bd1e229-67e3-4acd-8c95-e821a8812050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k" Oct 30 02:05:43.910034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.910012 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd1e229-67e3-4acd-8c95-e821a8812050-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81202a85-rjm6k\" (UID: \"9bd1e229-67e3-4acd-8c95-e821a8812050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k" Oct 30 02:05:43.920436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.920411 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn"] Oct 30 02:05:43.921062 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:05:43.921034 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b27c952_8035_4847_b243_4471250d3d09.slice/crio-d64dd504704dc4fa4a09b68354fa322f86c309aadbcc672b1a842a3e0d5a8b86 WatchSource:0}: Error finding container d64dd504704dc4fa4a09b68354fa322f86c309aadbcc672b1a842a3e0d5a8b86: Status 404 returned error can't find the container with id d64dd504704dc4fa4a09b68354fa322f86c309aadbcc672b1a842a3e0d5a8b86 Oct 30 02:05:43.923970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.923934 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:05:43.924044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.924022 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:05:43.924100 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.924065 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:05:43.929251 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:43.929229 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s27j\" (UniqueName: \"kubernetes.io/projected/9bd1e229-67e3-4acd-8c95-e821a8812050-kube-api-access-9s27j\") pod \"maas-api-revoke-sub-81202a85-rjm6k\" (UID: \"9bd1e229-67e3-4acd-8c95-e821a8812050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k" Oct 30 02:05:44.040036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:44.039996 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k" Oct 30 02:05:44.132009 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:44.131976 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:44.175477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:44.175425 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" event={"ID":"2b27c952-8035-4847-b243-4471250d3d09","Type":"ContainerStarted","Data":"ba8d5fa15cf551963cebbe78d9521409b26dff8203ffc421c1da7595761dff35"} Oct 30 02:05:44.175660 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:44.175500 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" event={"ID":"2b27c952-8035-4847-b243-4471250d3d09","Type":"ContainerStarted","Data":"d64dd504704dc4fa4a09b68354fa322f86c309aadbcc672b1a842a3e0d5a8b86"} Oct 30 02:05:44.236168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:44.234517 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" podStartSLOduration=1.234497293 podStartE2EDuration="1.234497293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:44.231179483 +0000 UTC m=+2173.809970564" watchObservedRunningTime="2026-10-30 02:05:44.234497293 +0000 UTC m=+2173.813288375" Oct 30 02:05:44.252147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:44.251510 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:44.355285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:44.355251 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k"] Oct 30 02:05:44.359304 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:05:44.359274 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd1e229_67e3_4acd_8c95_e821a8812050.slice/crio-948d1b65beca87d73331b3fbaa445aa8f807615d0266a7cd6b7d2f51421eac7d WatchSource:0}: Error finding container 948d1b65beca87d73331b3fbaa445aa8f807615d0266a7cd6b7d2f51421eac7d: Status 404 returned error can't find the container with id 948d1b65beca87d73331b3fbaa445aa8f807615d0266a7cd6b7d2f51421eac7d Oct 30 02:05:44.591789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:44.591743 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:44.594490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:44.594456 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 02:05:44.594688 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:44.594520 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" podUID="2b27c952-8035-4847-b243-4471250d3d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 02:05:44.863462 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:44.863334 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:45.185918 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:45.185873 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k" event={"ID":"9bd1e229-67e3-4acd-8c95-e821a8812050","Type":"ContainerStarted","Data":"1f4b6bc6a74a9172ec1c03eec1b251995884579951c6baa7bce2366ce5aa17b8"} Oct 30 02:05:45.186098 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:45.185973 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k" event={"ID":"9bd1e229-67e3-4acd-8c95-e821a8812050","Type":"ContainerStarted","Data":"948d1b65beca87d73331b3fbaa445aa8f807615d0266a7cd6b7d2f51421eac7d"} Oct 30 02:05:45.454205 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:45.429695 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:45.543751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:45.543714 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:45.592130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:45.592062 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 02:05:45.592289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:45.592163 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" podUID="2b27c952-8035-4847-b243-4471250d3d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 02:05:46.041593 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:46.041555 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:46.194603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:46.194552 5998 generic.go:356] "Generic (PLEG): container finished" podID="9bd1e229-67e3-4acd-8c95-e821a8812050" containerID="1f4b6bc6a74a9172ec1c03eec1b251995884579951c6baa7bce2366ce5aa17b8" exitCode=0 Oct 30 02:05:46.195493 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:46.195411 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k" event={"ID":"9bd1e229-67e3-4acd-8c95-e821a8812050","Type":"ContainerDied","Data":"1f4b6bc6a74a9172ec1c03eec1b251995884579951c6baa7bce2366ce5aa17b8"} Oct 30 02:05:46.592144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:46.592111 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 02:05:46.592546 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:46.592171 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" podUID="2b27c952-8035-4847-b243-4471250d3d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 02:05:47.247141 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:47.244369 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:47.549725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:47.549691 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k" Oct 30 02:05:47.591870 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:47.591831 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 02:05:47.592068 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:47.591884 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" podUID="2b27c952-8035-4847-b243-4471250d3d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 02:05:47.686152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:47.686098 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd1e229-67e3-4acd-8c95-e821a8812050-kube-api-access-9s27j\" (UniqueName: \"kubernetes.io/projected/9bd1e229-67e3-4acd-8c95-e821a8812050-kube-api-access-9s27j\") pod \"9bd1e229-67e3-4acd-8c95-e821a8812050\" (UID: \"9bd1e229-67e3-4acd-8c95-e821a8812050\") " Oct 30 02:05:47.686619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:47.686246 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd1e229-67e3-4acd-8c95-e821a8812050-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1e229-67e3-4acd-8c95-e821a8812050-maas-api-openshift-ca\") pod \"9bd1e229-67e3-4acd-8c95-e821a8812050\" (UID: \"9bd1e229-67e3-4acd-8c95-e821a8812050\") " Oct 30 02:05:47.686619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:47.686294 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bd1e229-67e3-4acd-8c95-e821a8812050-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd1e229-67e3-4acd-8c95-e821a8812050-maas-api-xks-ca\") pod \"9bd1e229-67e3-4acd-8c95-e821a8812050\" (UID: \"9bd1e229-67e3-4acd-8c95-e821a8812050\") " Oct 30 02:05:47.686619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:47.686598 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd1e229-67e3-4acd-8c95-e821a8812050-maas-api-openshift-ca" pod "9bd1e229-67e3-4acd-8c95-e821a8812050" (UID: "9bd1e229-67e3-4acd-8c95-e821a8812050"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:47.686939 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:47.686915 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1e229-67e3-4acd-8c95-e821a8812050-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:47.688655 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:47.688628 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd1e229-67e3-4acd-8c95-e821a8812050-kube-api-access-9s27j" pod "9bd1e229-67e3-4acd-8c95-e821a8812050" (UID: "9bd1e229-67e3-4acd-8c95-e821a8812050"). InnerVolumeSpecName "kube-api-access-9s27j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:47.688855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:47.688834 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bd1e229-67e3-4acd-8c95-e821a8812050-maas-api-xks-ca" pod "9bd1e229-67e3-4acd-8c95-e821a8812050" (UID: "9bd1e229-67e3-4acd-8c95-e821a8812050"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:47.788877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:47.788798 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd1e229-67e3-4acd-8c95-e821a8812050-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:47.788877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:47.788830 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s27j\" (UniqueName: \"kubernetes.io/projected/9bd1e229-67e3-4acd-8c95-e821a8812050-kube-api-access-9s27j\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:48.251241 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:48.251210 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k" Oct 30 02:05:48.251494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:48.251467 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k" event={"ID":"9bd1e229-67e3-4acd-8c95-e821a8812050","Type":"ContainerDied","Data":"948d1b65beca87d73331b3fbaa445aa8f807615d0266a7cd6b7d2f51421eac7d"} Oct 30 02:05:48.251614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:48.251599 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="948d1b65beca87d73331b3fbaa445aa8f807615d0266a7cd6b7d2f51421eac7d" Oct 30 02:05:48.592219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:48.592140 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 02:05:48.592219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:48.592206 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" podUID="2b27c952-8035-4847-b243-4471250d3d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 02:05:49.085149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:49.085116 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:49.247114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:49.246214 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:05:49.592084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:49.592050 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 02:05:49.592376 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:49.592133 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" podUID="2b27c952-8035-4847-b243-4471250d3d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 02:05:50.592520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:50.592474 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 30 02:05:50.593050 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:50.592561 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" podUID="2b27c952-8035-4847-b243-4471250d3d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 30 02:05:51.886116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:51.884247 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:52.115113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:52.094184 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 02:05:52.116743 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:52.115580 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:52.177236 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:52.176943 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:52.192769 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:52.192677 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:52.215491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:52.212208 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:52.215491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:52.215156 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:52.242048 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:52.241267 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:52.243178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:52.243144 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:52.475304 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:52.475192 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:52.477996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:52.477962 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:52.659308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:52.658276 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:05:52.659308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:52.658347 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" podUID="2b27c952-8035-4847-b243-4471250d3d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:52.894170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:52.885080 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:52.894170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:52.885811 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 02:05:52.992341 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:52.992312 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 02:05:53.086581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.069555 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:53.277128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.274056 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz"] Oct 30 02:05:53.277128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.274299 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" podUID="1bfb5bd1-c7a6-4760-b8b9-c26e06560652" containerName="discovery" containerID="cri-o://05b9eb9b80e37f987115bd3a54424b6abecd9f024a51f800160256d8725b3ec8" gracePeriod=30 Oct 30 02:05:53.473245 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.473156 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:53.630175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.626511 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:53.643026 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.642985 5998 generic.go:356] "Generic (PLEG): container finished" podID="1bfb5bd1-c7a6-4760-b8b9-c26e06560652" containerID="05b9eb9b80e37f987115bd3a54424b6abecd9f024a51f800160256d8725b3ec8" exitCode=0 Oct 30 02:05:53.644196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.644163 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" event={"ID":"1bfb5bd1-c7a6-4760-b8b9-c26e06560652","Type":"ContainerDied","Data":"05b9eb9b80e37f987115bd3a54424b6abecd9f024a51f800160256d8725b3ec8"} Oct 30 02:05:53.663501 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.663465 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:05:53.663790 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.663762 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" podUID="2b27c952-8035-4847-b243-4471250d3d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:53.875890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.875861 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 02:05:53.879878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.879851 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-csr-ca-configmap\") pod \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " Oct 30 02:05:53.879990 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.879940 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-cacerts\" (UniqueName: \"kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-cacerts\") pod \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " Oct 30 02:05:53.880054 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.880003 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-csr-dns-cert\") pod \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " Oct 30 02:05:53.880142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.880109 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-local-certs\") pod \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " Oct 30 02:05:53.880253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.880213 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-kubeconfig\") pod \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " Oct 30 02:05:53.880324 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.880265 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-kube-api-access-nv6zk\" (UniqueName: \"kubernetes.io/projected/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-kube-api-access-nv6zk\") pod \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " Oct 30 02:05:53.880430 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.880407 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-csr-ca-configmap" pod "1bfb5bd1-c7a6-4760-b8b9-c26e06560652" (UID: "1bfb5bd1-c7a6-4760-b8b9-c26e06560652"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:53.880750 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.880733 5998 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-csr-ca-configmap\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:53.883400 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.883365 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-csr-dns-cert" pod "1bfb5bd1-c7a6-4760-b8b9-c26e06560652" (UID: "1bfb5bd1-c7a6-4760-b8b9-c26e06560652"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:53.883789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.883749 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-kubeconfig" pod "1bfb5bd1-c7a6-4760-b8b9-c26e06560652" (UID: "1bfb5bd1-c7a6-4760-b8b9-c26e06560652"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:53.883979 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.883959 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-cacerts" pod "1bfb5bd1-c7a6-4760-b8b9-c26e06560652" (UID: "1bfb5bd1-c7a6-4760-b8b9-c26e06560652"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:53.884134 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.884113 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-kube-api-access-nv6zk" pod "1bfb5bd1-c7a6-4760-b8b9-c26e06560652" (UID: "1bfb5bd1-c7a6-4760-b8b9-c26e06560652"). InnerVolumeSpecName "kube-api-access-nv6zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:53.884921 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.884898 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-local-certs" pod "1bfb5bd1-c7a6-4760-b8b9-c26e06560652" (UID: "1bfb5bd1-c7a6-4760-b8b9-c26e06560652"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:05:53.981241 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.981208 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-token\" (UniqueName: \"kubernetes.io/projected/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-token\") pod \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\" (UID: \"1bfb5bd1-c7a6-4760-b8b9-c26e06560652\") " Oct 30 02:05:53.981829 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.981807 5998 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-cacerts\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:53.981938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.981833 5998 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-csr-dns-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:53.981938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.981846 5998 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-local-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:53.981938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.981859 5998 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-kubeconfig\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:53.981938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.981872 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv6zk\" (UniqueName: \"kubernetes.io/projected/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-kube-api-access-nv6zk\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:53.983928 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:53.983904 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-token" pod "1bfb5bd1-c7a6-4760-b8b9-c26e06560652" (UID: "1bfb5bd1-c7a6-4760-b8b9-c26e06560652"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:54.082794 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.082757 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bfb5bd1-c7a6-4760-b8b9-c26e06560652-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:54.235115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.234116 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9"] Oct 30 02:05:54.235115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.234903 5998 cpu_manager.go:414] "containerMap: removing container" podUID="9bd1e229-67e3-4acd-8c95-e821a8812050" containerName="revoke-keys" Oct 30 02:05:54.235115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.234919 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd1e229-67e3-4acd-8c95-e821a8812050" containerName="revoke-keys" Oct 30 02:05:54.235115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.234948 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1bfb5bd1-c7a6-4760-b8b9-c26e06560652" containerName="discovery" Oct 30 02:05:54.235115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.234956 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bfb5bd1-c7a6-4760-b8b9-c26e06560652" containerName="discovery" Oct 30 02:05:54.235115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.235071 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd1e229-67e3-4acd-8c95-e821a8812050" containerName="revoke-keys" Oct 30 02:05:54.235115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.235084 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bfb5bd1-c7a6-4760-b8b9-c26e06560652" containerName="discovery" Oct 30 02:05:54.249115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.239268 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9" Oct 30 02:05:54.253100 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.253011 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:05:54.253209 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.253166 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:05:54.253274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.253226 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:05:54.253340 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.253288 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:05:54.263962 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.263924 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9"] Oct 30 02:05:54.284161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.284128 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a844676-c921-41c0-a1f7-12d5c962b73e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-147ea4a5-wbtp9\" (UID: \"1a844676-c921-41c0-a1f7-12d5c962b73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9" Oct 30 02:05:54.284316 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.284190 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a844676-c921-41c0-a1f7-12d5c962b73e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-147ea4a5-wbtp9\" (UID: \"1a844676-c921-41c0-a1f7-12d5c962b73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9" Oct 30 02:05:54.284409 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.284383 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4jr2\" (UniqueName: \"kubernetes.io/projected/1a844676-c921-41c0-a1f7-12d5c962b73e-kube-api-access-n4jr2\") pod \"maas-api-revoke-sub-147ea4a5-wbtp9\" (UID: \"1a844676-c921-41c0-a1f7-12d5c962b73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9" Oct 30 02:05:54.385343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.385307 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a844676-c921-41c0-a1f7-12d5c962b73e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-147ea4a5-wbtp9\" (UID: \"1a844676-c921-41c0-a1f7-12d5c962b73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9" Oct 30 02:05:54.385501 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.385452 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a844676-c921-41c0-a1f7-12d5c962b73e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-147ea4a5-wbtp9\" (UID: \"1a844676-c921-41c0-a1f7-12d5c962b73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9" Oct 30 02:05:54.385578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.385564 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4jr2\" (UniqueName: \"kubernetes.io/projected/1a844676-c921-41c0-a1f7-12d5c962b73e-kube-api-access-n4jr2\") pod \"maas-api-revoke-sub-147ea4a5-wbtp9\" (UID: \"1a844676-c921-41c0-a1f7-12d5c962b73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9" Oct 30 02:05:54.385944 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.385917 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a844676-c921-41c0-a1f7-12d5c962b73e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-147ea4a5-wbtp9\" (UID: \"1a844676-c921-41c0-a1f7-12d5c962b73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9" Oct 30 02:05:54.387865 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.387846 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a844676-c921-41c0-a1f7-12d5c962b73e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-147ea4a5-wbtp9\" (UID: \"1a844676-c921-41c0-a1f7-12d5c962b73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9" Oct 30 02:05:54.389688 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.389665 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:54.411111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.406985 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4jr2\" (UniqueName: \"kubernetes.io/projected/1a844676-c921-41c0-a1f7-12d5c962b73e-kube-api-access-n4jr2\") pod \"maas-api-revoke-sub-147ea4a5-wbtp9\" (UID: \"1a844676-c921-41c0-a1f7-12d5c962b73e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9" Oct 30 02:05:54.563773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.563724 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9" Oct 30 02:05:54.668419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.668375 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" event={"ID":"1bfb5bd1-c7a6-4760-b8b9-c26e06560652","Type":"ContainerDied","Data":"8c038d10d52ca7a847ceb36d1ff4853d325c50854ed789e0901ff901b2a81df2"} Oct 30 02:05:54.668611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.668454 5998 scope.go:122] "RemoveContainer" containerID="05b9eb9b80e37f987115bd3a54424b6abecd9f024a51f800160256d8725b3ec8" Oct 30 02:05:54.668690 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.668631 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz" Oct 30 02:05:54.706153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.705237 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:05:54.706153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.705364 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" podUID="2b27c952-8035-4847-b243-4471250d3d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:05:54.721951 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.721913 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz"] Oct 30 02:05:54.748193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.747626 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bt6gz"] Oct 30 02:05:54.864365 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.864283 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:54.864686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.864663 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:54.875514 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.875448 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9"] Oct 30 02:05:54.877962 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:05:54.877936 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a844676_c921_41c0_a1f7_12d5c962b73e.slice/crio-de566b2c1c0d57b6762bddf0298979d7c521bee62de2edb056fa4b57ee26345c WatchSource:0}: Error finding container de566b2c1c0d57b6762bddf0298979d7c521bee62de2edb056fa4b57ee26345c: Status 404 returned error can't find the container with id de566b2c1c0d57b6762bddf0298979d7c521bee62de2edb056fa4b57ee26345c Oct 30 02:05:54.909011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.908973 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:05:54.932328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:54.932291 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:55.233312 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:05:55.233278 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:05:55.406448 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:55.406402 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:55.592220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:55.591926 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:55.678180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:55.678142 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9" event={"ID":"1a844676-c921-41c0-a1f7-12d5c962b73e","Type":"ContainerStarted","Data":"8c6c248a5bddfe7ee9f7c77488c238bab6957e1c7d3d0588b2965e303e6d52c9"} Oct 30 02:05:55.678363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:55.678225 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9" event={"ID":"1a844676-c921-41c0-a1f7-12d5c962b73e","Type":"ContainerStarted","Data":"de566b2c1c0d57b6762bddf0298979d7c521bee62de2edb056fa4b57ee26345c"} Oct 30 02:05:55.721422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:55.721353 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9" podStartSLOduration=1.721331417 podStartE2EDuration="1.721331417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:55.715198057 +0000 UTC m=+2185.293989141" watchObservedRunningTime="2026-10-30 02:05:55.721331417 +0000 UTC m=+2185.300122499" Oct 30 02:05:56.087546 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:56.087502 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:56.543127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:56.543064 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bfb5bd1-c7a6-4760-b8b9-c26e06560652" path="/var/lib/kubelet/pods/1bfb5bd1-c7a6-4760-b8b9-c26e06560652/volumes" Oct 30 02:05:56.688927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:56.688887 5998 generic.go:356] "Generic (PLEG): container finished" podID="1a844676-c921-41c0-a1f7-12d5c962b73e" containerID="8c6c248a5bddfe7ee9f7c77488c238bab6957e1c7d3d0588b2965e303e6d52c9" exitCode=0 Oct 30 02:05:56.689120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:56.688981 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9" event={"ID":"1a844676-c921-41c0-a1f7-12d5c962b73e","Type":"ContainerDied","Data":"8c6c248a5bddfe7ee9f7c77488c238bab6957e1c7d3d0588b2965e303e6d52c9"} Oct 30 02:05:57.662447 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:57.662405 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:57.887572 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:57.887544 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9" Oct 30 02:05:57.942285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:57.942190 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a844676-c921-41c0-a1f7-12d5c962b73e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a844676-c921-41c0-a1f7-12d5c962b73e-maas-api-openshift-ca\") pod \"1a844676-c921-41c0-a1f7-12d5c962b73e\" (UID: \"1a844676-c921-41c0-a1f7-12d5c962b73e\") " Oct 30 02:05:57.942285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:57.942260 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a844676-c921-41c0-a1f7-12d5c962b73e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a844676-c921-41c0-a1f7-12d5c962b73e-maas-api-xks-ca\") pod \"1a844676-c921-41c0-a1f7-12d5c962b73e\" (UID: \"1a844676-c921-41c0-a1f7-12d5c962b73e\") " Oct 30 02:05:57.942555 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:57.942532 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a844676-c921-41c0-a1f7-12d5c962b73e-kube-api-access-n4jr2\" (UniqueName: \"kubernetes.io/projected/1a844676-c921-41c0-a1f7-12d5c962b73e-kube-api-access-n4jr2\") pod \"1a844676-c921-41c0-a1f7-12d5c962b73e\" (UID: \"1a844676-c921-41c0-a1f7-12d5c962b73e\") " Oct 30 02:05:57.943823 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:57.943789 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a844676-c921-41c0-a1f7-12d5c962b73e-maas-api-openshift-ca" pod "1a844676-c921-41c0-a1f7-12d5c962b73e" (UID: "1a844676-c921-41c0-a1f7-12d5c962b73e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:05:57.944942 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:57.944918 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a844676-c921-41c0-a1f7-12d5c962b73e-maas-api-xks-ca" pod "1a844676-c921-41c0-a1f7-12d5c962b73e" (UID: "1a844676-c921-41c0-a1f7-12d5c962b73e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:05:57.945278 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:57.945237 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a844676-c921-41c0-a1f7-12d5c962b73e-kube-api-access-n4jr2" pod "1a844676-c921-41c0-a1f7-12d5c962b73e" (UID: "1a844676-c921-41c0-a1f7-12d5c962b73e"). InnerVolumeSpecName "kube-api-access-n4jr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:05:58.044410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:58.044369 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4jr2\" (UniqueName: \"kubernetes.io/projected/1a844676-c921-41c0-a1f7-12d5c962b73e-kube-api-access-n4jr2\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:58.044410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:58.044403 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a844676-c921-41c0-a1f7-12d5c962b73e-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:58.044410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:58.044414 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a844676-c921-41c0-a1f7-12d5c962b73e-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:05:58.705876 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:58.705840 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9" Oct 30 02:05:58.706432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:58.705869 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9" event={"ID":"1a844676-c921-41c0-a1f7-12d5c962b73e","Type":"ContainerDied","Data":"de566b2c1c0d57b6762bddf0298979d7c521bee62de2edb056fa4b57ee26345c"} Oct 30 02:05:58.706432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:58.705927 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de566b2c1c0d57b6762bddf0298979d7c521bee62de2edb056fa4b57ee26345c" Oct 30 02:05:59.397211 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:59.397168 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:05:59.723149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:05:59.723109 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:01.072066 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:01.072032 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:01.179101 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:01.179053 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:01.290684 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:01.290649 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:03.019255 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:03.019218 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:03.127701 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:03.127664 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:04.580156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:04.580120 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:05.883234 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:05.883195 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:06.054403 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:06.054366 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:06.161991 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:06.161957 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:06.654792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:06.654704 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:08.379806 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:08.379759 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:09.962172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:09.962136 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:10.075613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:10.075577 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:10.592667 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:10.592631 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:11.007867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.007823 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:11.580600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.580565 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w"] Oct 30 02:06:11.581150 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.581135 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1a844676-c921-41c0-a1f7-12d5c962b73e" containerName="revoke-keys" Oct 30 02:06:11.581150 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.581150 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a844676-c921-41c0-a1f7-12d5c962b73e" containerName="revoke-keys" Oct 30 02:06:11.581258 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.581238 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a844676-c921-41c0-a1f7-12d5c962b73e" containerName="revoke-keys" Oct 30 02:06:11.586060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.586038 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w" Oct 30 02:06:11.599240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.599215 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:06:11.609408 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.609380 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:06:11.609565 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.609432 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:06:11.609565 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.609455 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:06:11.614785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.614758 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w"] Oct 30 02:06:11.655944 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.655905 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b5bz\" (UniqueName: \"kubernetes.io/projected/8ee1fda9-fa93-431b-a16b-8d0a2c520811-kube-api-access-6b5bz\") pod \"maas-api-revoke-sub-96fdbffa-g7v6w\" (UID: \"8ee1fda9-fa93-431b-a16b-8d0a2c520811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w" Oct 30 02:06:11.656267 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.656235 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee1fda9-fa93-431b-a16b-8d0a2c520811-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96fdbffa-g7v6w\" (UID: \"8ee1fda9-fa93-431b-a16b-8d0a2c520811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w" Oct 30 02:06:11.656434 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.656410 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee1fda9-fa93-431b-a16b-8d0a2c520811-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96fdbffa-g7v6w\" (UID: \"8ee1fda9-fa93-431b-a16b-8d0a2c520811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w" Oct 30 02:06:11.757881 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.757839 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b5bz\" (UniqueName: \"kubernetes.io/projected/8ee1fda9-fa93-431b-a16b-8d0a2c520811-kube-api-access-6b5bz\") pod \"maas-api-revoke-sub-96fdbffa-g7v6w\" (UID: \"8ee1fda9-fa93-431b-a16b-8d0a2c520811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w" Oct 30 02:06:11.758083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.757900 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee1fda9-fa93-431b-a16b-8d0a2c520811-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96fdbffa-g7v6w\" (UID: \"8ee1fda9-fa93-431b-a16b-8d0a2c520811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w" Oct 30 02:06:11.758083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.758067 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee1fda9-fa93-431b-a16b-8d0a2c520811-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96fdbffa-g7v6w\" (UID: \"8ee1fda9-fa93-431b-a16b-8d0a2c520811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w" Oct 30 02:06:11.758754 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.758731 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee1fda9-fa93-431b-a16b-8d0a2c520811-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96fdbffa-g7v6w\" (UID: \"8ee1fda9-fa93-431b-a16b-8d0a2c520811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w" Oct 30 02:06:11.760459 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.760434 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee1fda9-fa93-431b-a16b-8d0a2c520811-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96fdbffa-g7v6w\" (UID: \"8ee1fda9-fa93-431b-a16b-8d0a2c520811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w" Oct 30 02:06:11.779689 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.779661 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b5bz\" (UniqueName: \"kubernetes.io/projected/8ee1fda9-fa93-431b-a16b-8d0a2c520811-kube-api-access-6b5bz\") pod \"maas-api-revoke-sub-96fdbffa-g7v6w\" (UID: \"8ee1fda9-fa93-431b-a16b-8d0a2c520811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w" Oct 30 02:06:11.863676 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.863591 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:11.899596 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:11.899559 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w" Oct 30 02:06:12.253410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:12.253386 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w"] Oct 30 02:06:12.255789 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:06:12.255759 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee1fda9_fa93_431b_a16b_8d0a2c520811.slice/crio-3b32df14206300aee1fafad7c6d7789cfa19a0ebe7eac29fadea32f3e23e1281 WatchSource:0}: Error finding container 3b32df14206300aee1fafad7c6d7789cfa19a0ebe7eac29fadea32f3e23e1281: Status 404 returned error can't find the container with id 3b32df14206300aee1fafad7c6d7789cfa19a0ebe7eac29fadea32f3e23e1281 Oct 30 02:06:12.808471 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:12.808361 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w" event={"ID":"8ee1fda9-fa93-431b-a16b-8d0a2c520811","Type":"ContainerStarted","Data":"1b0aa83df763c0c5b32fe3271a3dda6bad95836ef4f38b8c7b7d525bf3d9ca53"} Oct 30 02:06:12.808471 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:12.808425 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w" event={"ID":"8ee1fda9-fa93-431b-a16b-8d0a2c520811","Type":"ContainerStarted","Data":"3b32df14206300aee1fafad7c6d7789cfa19a0ebe7eac29fadea32f3e23e1281"} Oct 30 02:06:12.848420 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:12.848350 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w" podStartSLOduration=1.848330127 podStartE2EDuration="1.848330127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:12.845182594 +0000 UTC m=+2202.423973676" watchObservedRunningTime="2026-10-30 02:06:12.848330127 +0000 UTC m=+2202.427121209" Oct 30 02:06:13.537718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:13.537629 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:13.654767 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:13.654733 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:13.818928 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:13.818836 5998 generic.go:356] "Generic (PLEG): container finished" podID="8ee1fda9-fa93-431b-a16b-8d0a2c520811" containerID="1b0aa83df763c0c5b32fe3271a3dda6bad95836ef4f38b8c7b7d525bf3d9ca53" exitCode=0 Oct 30 02:06:13.819066 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:13.818922 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w" event={"ID":"8ee1fda9-fa93-431b-a16b-8d0a2c520811","Type":"ContainerDied","Data":"1b0aa83df763c0c5b32fe3271a3dda6bad95836ef4f38b8c7b7d525bf3d9ca53"} Oct 30 02:06:14.987979 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:14.987955 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w" Oct 30 02:06:15.109359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:15.109259 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee1fda9-fa93-431b-a16b-8d0a2c520811-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee1fda9-fa93-431b-a16b-8d0a2c520811-maas-api-openshift-ca\") pod \"8ee1fda9-fa93-431b-a16b-8d0a2c520811\" (UID: \"8ee1fda9-fa93-431b-a16b-8d0a2c520811\") " Oct 30 02:06:15.109548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:15.109377 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ee1fda9-fa93-431b-a16b-8d0a2c520811-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee1fda9-fa93-431b-a16b-8d0a2c520811-maas-api-xks-ca\") pod \"8ee1fda9-fa93-431b-a16b-8d0a2c520811\" (UID: \"8ee1fda9-fa93-431b-a16b-8d0a2c520811\") " Oct 30 02:06:15.109548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:15.109479 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee1fda9-fa93-431b-a16b-8d0a2c520811-kube-api-access-6b5bz\" (UniqueName: \"kubernetes.io/projected/8ee1fda9-fa93-431b-a16b-8d0a2c520811-kube-api-access-6b5bz\") pod \"8ee1fda9-fa93-431b-a16b-8d0a2c520811\" (UID: \"8ee1fda9-fa93-431b-a16b-8d0a2c520811\") " Oct 30 02:06:15.109671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:15.109624 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee1fda9-fa93-431b-a16b-8d0a2c520811-maas-api-openshift-ca" pod "8ee1fda9-fa93-431b-a16b-8d0a2c520811" (UID: "8ee1fda9-fa93-431b-a16b-8d0a2c520811"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:15.110174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:15.110148 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee1fda9-fa93-431b-a16b-8d0a2c520811-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:15.111765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:15.111731 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee1fda9-fa93-431b-a16b-8d0a2c520811-kube-api-access-6b5bz" pod "8ee1fda9-fa93-431b-a16b-8d0a2c520811" (UID: "8ee1fda9-fa93-431b-a16b-8d0a2c520811"). InnerVolumeSpecName "kube-api-access-6b5bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:15.111863 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:15.111739 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee1fda9-fa93-431b-a16b-8d0a2c520811-maas-api-xks-ca" pod "8ee1fda9-fa93-431b-a16b-8d0a2c520811" (UID: "8ee1fda9-fa93-431b-a16b-8d0a2c520811"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:06:15.211198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:15.211162 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee1fda9-fa93-431b-a16b-8d0a2c520811-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:15.211198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:15.211197 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b5bz\" (UniqueName: \"kubernetes.io/projected/8ee1fda9-fa93-431b-a16b-8d0a2c520811-kube-api-access-6b5bz\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:15.264843 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:15.264810 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:15.582563 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:15.582531 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:06:15.836689 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:15.836597 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w" Oct 30 02:06:15.836689 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:15.836604 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w" event={"ID":"8ee1fda9-fa93-431b-a16b-8d0a2c520811","Type":"ContainerDied","Data":"3b32df14206300aee1fafad7c6d7789cfa19a0ebe7eac29fadea32f3e23e1281"} Oct 30 02:06:15.836928 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:15.836725 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b32df14206300aee1fafad7c6d7789cfa19a0ebe7eac29fadea32f3e23e1281" Oct 30 02:06:17.194118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:17.192984 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:19.002715 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.002680 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959"] Oct 30 02:06:19.003350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.003331 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8ee1fda9-fa93-431b-a16b-8d0a2c520811" containerName="revoke-keys" Oct 30 02:06:19.003402 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.003352 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee1fda9-fa93-431b-a16b-8d0a2c520811" containerName="revoke-keys" Oct 30 02:06:19.003506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.003493 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee1fda9-fa93-431b-a16b-8d0a2c520811" containerName="revoke-keys" Oct 30 02:06:19.008482 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.008460 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959" Oct 30 02:06:19.024553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.024521 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:06:19.024686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.024581 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:06:19.028615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.028587 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959"] Oct 30 02:06:19.034793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.034767 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:06:19.034924 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.034769 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:06:19.137071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.137035 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8tf9\" (UniqueName: \"kubernetes.io/projected/c2f3c8d5-6721-4780-a6dc-9d66c4785930-kube-api-access-g8tf9\") pod \"maas-api-revoke-sub-32e3afef-q7959\" (UID: \"c2f3c8d5-6721-4780-a6dc-9d66c4785930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959" Oct 30 02:06:19.137433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.137403 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f3c8d5-6721-4780-a6dc-9d66c4785930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32e3afef-q7959\" (UID: \"c2f3c8d5-6721-4780-a6dc-9d66c4785930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959" Oct 30 02:06:19.137511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.137458 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f3c8d5-6721-4780-a6dc-9d66c4785930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32e3afef-q7959\" (UID: \"c2f3c8d5-6721-4780-a6dc-9d66c4785930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959" Oct 30 02:06:19.239183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.239142 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f3c8d5-6721-4780-a6dc-9d66c4785930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32e3afef-q7959\" (UID: \"c2f3c8d5-6721-4780-a6dc-9d66c4785930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959" Oct 30 02:06:19.239406 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.239198 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f3c8d5-6721-4780-a6dc-9d66c4785930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32e3afef-q7959\" (UID: \"c2f3c8d5-6721-4780-a6dc-9d66c4785930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959" Oct 30 02:06:19.239406 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.239260 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8tf9\" (UniqueName: \"kubernetes.io/projected/c2f3c8d5-6721-4780-a6dc-9d66c4785930-kube-api-access-g8tf9\") pod \"maas-api-revoke-sub-32e3afef-q7959\" (UID: \"c2f3c8d5-6721-4780-a6dc-9d66c4785930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959" Oct 30 02:06:19.239963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.239932 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f3c8d5-6721-4780-a6dc-9d66c4785930-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32e3afef-q7959\" (UID: \"c2f3c8d5-6721-4780-a6dc-9d66c4785930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959" Oct 30 02:06:19.242420 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.242375 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f3c8d5-6721-4780-a6dc-9d66c4785930-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32e3afef-q7959\" (UID: \"c2f3c8d5-6721-4780-a6dc-9d66c4785930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959" Oct 30 02:06:19.261160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.261075 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8tf9\" (UniqueName: \"kubernetes.io/projected/c2f3c8d5-6721-4780-a6dc-9d66c4785930-kube-api-access-g8tf9\") pod \"maas-api-revoke-sub-32e3afef-q7959\" (UID: \"c2f3c8d5-6721-4780-a6dc-9d66c4785930\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959" Oct 30 02:06:19.322581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.322548 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959" Oct 30 02:06:19.397134 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.393639 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:19.636645 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.636611 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959"] Oct 30 02:06:19.638128 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:06:19.638066 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f3c8d5_6721_4780_a6dc_9d66c4785930.slice/crio-625131ad8210fb029b7b79704501bf776193cd30df04aa3f7219d026c442fe5e WatchSource:0}: Error finding container 625131ad8210fb029b7b79704501bf776193cd30df04aa3f7219d026c442fe5e: Status 404 returned error can't find the container with id 625131ad8210fb029b7b79704501bf776193cd30df04aa3f7219d026c442fe5e Oct 30 02:06:19.927378 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.927344 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959" event={"ID":"c2f3c8d5-6721-4780-a6dc-9d66c4785930","Type":"ContainerStarted","Data":"8d7af1a7b2b710597fce498045890efa229ee54491a43d602d9d68ee9d61dc78"} Oct 30 02:06:19.927488 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:19.927405 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959" event={"ID":"c2f3c8d5-6721-4780-a6dc-9d66c4785930","Type":"ContainerStarted","Data":"625131ad8210fb029b7b79704501bf776193cd30df04aa3f7219d026c442fe5e"} Oct 30 02:06:20.511718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:20.511651 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959" podStartSLOduration=2.511631444 podStartE2EDuration="2.511631444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:19.969655055 +0000 UTC m=+2209.548446136" watchObservedRunningTime="2026-10-30 02:06:20.511631444 +0000 UTC m=+2210.090422528" Oct 30 02:06:20.512312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:20.512292 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p"] Oct 30 02:06:20.517324 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:20.517302 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p" Oct 30 02:06:20.543532 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:20.543487 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p"] Oct 30 02:06:20.659626 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:20.659591 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bdd3394-1315-4d86-988f-ab7fafafa831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-681f3da8-99c6p\" (UID: \"7bdd3394-1315-4d86-988f-ab7fafafa831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p" Oct 30 02:06:20.659920 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:20.659795 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bdd3394-1315-4d86-988f-ab7fafafa831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-681f3da8-99c6p\" (UID: \"7bdd3394-1315-4d86-988f-ab7fafafa831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p" Oct 30 02:06:20.660232 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:20.660200 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwl4n\" (UniqueName: \"kubernetes.io/projected/7bdd3394-1315-4d86-988f-ab7fafafa831-kube-api-access-gwl4n\") pod \"maas-api-revoke-sub-681f3da8-99c6p\" (UID: \"7bdd3394-1315-4d86-988f-ab7fafafa831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p" Oct 30 02:06:20.761656 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:20.761568 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bdd3394-1315-4d86-988f-ab7fafafa831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-681f3da8-99c6p\" (UID: \"7bdd3394-1315-4d86-988f-ab7fafafa831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p" Oct 30 02:06:20.761807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:20.761688 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwl4n\" (UniqueName: \"kubernetes.io/projected/7bdd3394-1315-4d86-988f-ab7fafafa831-kube-api-access-gwl4n\") pod \"maas-api-revoke-sub-681f3da8-99c6p\" (UID: \"7bdd3394-1315-4d86-988f-ab7fafafa831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p" Oct 30 02:06:20.761807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:20.761713 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bdd3394-1315-4d86-988f-ab7fafafa831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-681f3da8-99c6p\" (UID: \"7bdd3394-1315-4d86-988f-ab7fafafa831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p" Oct 30 02:06:20.762306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:20.762268 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bdd3394-1315-4d86-988f-ab7fafafa831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-681f3da8-99c6p\" (UID: \"7bdd3394-1315-4d86-988f-ab7fafafa831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p" Oct 30 02:06:20.763964 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:20.763944 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bdd3394-1315-4d86-988f-ab7fafafa831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-681f3da8-99c6p\" (UID: \"7bdd3394-1315-4d86-988f-ab7fafafa831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p" Oct 30 02:06:20.784009 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:20.783982 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwl4n\" (UniqueName: \"kubernetes.io/projected/7bdd3394-1315-4d86-988f-ab7fafafa831-kube-api-access-gwl4n\") pod \"maas-api-revoke-sub-681f3da8-99c6p\" (UID: \"7bdd3394-1315-4d86-988f-ab7fafafa831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p" Oct 30 02:06:20.872297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:20.872257 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p" Oct 30 02:06:20.953127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:20.950553 5998 generic.go:356] "Generic (PLEG): container finished" podID="c2f3c8d5-6721-4780-a6dc-9d66c4785930" containerID="8d7af1a7b2b710597fce498045890efa229ee54491a43d602d9d68ee9d61dc78" exitCode=0 Oct 30 02:06:20.953127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:20.950819 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959" event={"ID":"c2f3c8d5-6721-4780-a6dc-9d66c4785930","Type":"ContainerDied","Data":"8d7af1a7b2b710597fce498045890efa229ee54491a43d602d9d68ee9d61dc78"} Oct 30 02:06:21.091247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:21.091157 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:21.208477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:21.208444 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p"] Oct 30 02:06:21.208761 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:06:21.208728 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bdd3394_1315_4d86_988f_ab7fafafa831.slice/crio-fb01b828bcf648b5fee72a6baaa11a8ba59f119b398d763845f271be9aa12f31 WatchSource:0}: Error finding container fb01b828bcf648b5fee72a6baaa11a8ba59f119b398d763845f271be9aa12f31: Status 404 returned error can't find the container with id fb01b828bcf648b5fee72a6baaa11a8ba59f119b398d763845f271be9aa12f31 Oct 30 02:06:21.959770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:21.959735 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p" event={"ID":"7bdd3394-1315-4d86-988f-ab7fafafa831","Type":"ContainerStarted","Data":"9296e05d2e83f2170ea7fb49622fcb857d932200ec65d7477051b3fca4086f10"} Oct 30 02:06:21.960259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:21.959822 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p" event={"ID":"7bdd3394-1315-4d86-988f-ab7fafafa831","Type":"ContainerStarted","Data":"fb01b828bcf648b5fee72a6baaa11a8ba59f119b398d763845f271be9aa12f31"} Oct 30 02:06:22.009455 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:22.009392 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p" podStartSLOduration=2.009371391 podStartE2EDuration="2.009371391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:22.008524011 +0000 UTC m=+2211.587315094" watchObservedRunningTime="2026-10-30 02:06:22.009371391 +0000 UTC m=+2211.588162472" Oct 30 02:06:22.177191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:22.177164 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959" Oct 30 02:06:22.280178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:22.280075 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2f3c8d5-6721-4780-a6dc-9d66c4785930-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f3c8d5-6721-4780-a6dc-9d66c4785930-maas-api-openshift-ca\") pod \"c2f3c8d5-6721-4780-a6dc-9d66c4785930\" (UID: \"c2f3c8d5-6721-4780-a6dc-9d66c4785930\") " Oct 30 02:06:22.280347 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:22.280285 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f3c8d5-6721-4780-a6dc-9d66c4785930-kube-api-access-g8tf9\" (UniqueName: \"kubernetes.io/projected/c2f3c8d5-6721-4780-a6dc-9d66c4785930-kube-api-access-g8tf9\") pod \"c2f3c8d5-6721-4780-a6dc-9d66c4785930\" (UID: \"c2f3c8d5-6721-4780-a6dc-9d66c4785930\") " Oct 30 02:06:22.280347 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:22.280334 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2f3c8d5-6721-4780-a6dc-9d66c4785930-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f3c8d5-6721-4780-a6dc-9d66c4785930-maas-api-xks-ca\") pod \"c2f3c8d5-6721-4780-a6dc-9d66c4785930\" (UID: \"c2f3c8d5-6721-4780-a6dc-9d66c4785930\") " Oct 30 02:06:22.280469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:22.280439 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f3c8d5-6721-4780-a6dc-9d66c4785930-maas-api-openshift-ca" pod "c2f3c8d5-6721-4780-a6dc-9d66c4785930" (UID: "c2f3c8d5-6721-4780-a6dc-9d66c4785930"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:22.280792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:22.280774 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f3c8d5-6721-4780-a6dc-9d66c4785930-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:22.282344 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:22.282316 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f3c8d5-6721-4780-a6dc-9d66c4785930-kube-api-access-g8tf9" pod "c2f3c8d5-6721-4780-a6dc-9d66c4785930" (UID: "c2f3c8d5-6721-4780-a6dc-9d66c4785930"). InnerVolumeSpecName "kube-api-access-g8tf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:22.282442 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:22.282350 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2f3c8d5-6721-4780-a6dc-9d66c4785930-maas-api-xks-ca" pod "c2f3c8d5-6721-4780-a6dc-9d66c4785930" (UID: "c2f3c8d5-6721-4780-a6dc-9d66c4785930"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:06:22.382373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:22.382336 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8tf9\" (UniqueName: \"kubernetes.io/projected/c2f3c8d5-6721-4780-a6dc-9d66c4785930-kube-api-access-g8tf9\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:22.382373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:22.382367 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f3c8d5-6721-4780-a6dc-9d66c4785930-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:22.696831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:22.696796 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:22.968413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:22.968379 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959" Oct 30 02:06:22.968413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:22.968407 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959" event={"ID":"c2f3c8d5-6721-4780-a6dc-9d66c4785930","Type":"ContainerDied","Data":"625131ad8210fb029b7b79704501bf776193cd30df04aa3f7219d026c442fe5e"} Oct 30 02:06:22.968971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:22.968453 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="625131ad8210fb029b7b79704501bf776193cd30df04aa3f7219d026c442fe5e" Oct 30 02:06:22.970287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:22.970256 5998 generic.go:356] "Generic (PLEG): container finished" podID="7bdd3394-1315-4d86-988f-ab7fafafa831" containerID="9296e05d2e83f2170ea7fb49622fcb857d932200ec65d7477051b3fca4086f10" exitCode=0 Oct 30 02:06:22.970424 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:22.970290 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p" event={"ID":"7bdd3394-1315-4d86-988f-ab7fafafa831","Type":"ContainerDied","Data":"9296e05d2e83f2170ea7fb49622fcb857d932200ec65d7477051b3fca4086f10"} Oct 30 02:06:24.135125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:24.135074 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p" Oct 30 02:06:24.309384 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:24.309297 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bdd3394-1315-4d86-988f-ab7fafafa831-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bdd3394-1315-4d86-988f-ab7fafafa831-maas-api-xks-ca\") pod \"7bdd3394-1315-4d86-988f-ab7fafafa831\" (UID: \"7bdd3394-1315-4d86-988f-ab7fafafa831\") " Oct 30 02:06:24.309384 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:24.309349 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bdd3394-1315-4d86-988f-ab7fafafa831-kube-api-access-gwl4n\" (UniqueName: \"kubernetes.io/projected/7bdd3394-1315-4d86-988f-ab7fafafa831-kube-api-access-gwl4n\") pod \"7bdd3394-1315-4d86-988f-ab7fafafa831\" (UID: \"7bdd3394-1315-4d86-988f-ab7fafafa831\") " Oct 30 02:06:24.309610 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:24.309408 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bdd3394-1315-4d86-988f-ab7fafafa831-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bdd3394-1315-4d86-988f-ab7fafafa831-maas-api-openshift-ca\") pod \"7bdd3394-1315-4d86-988f-ab7fafafa831\" (UID: \"7bdd3394-1315-4d86-988f-ab7fafafa831\") " Oct 30 02:06:24.309927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:24.309892 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bdd3394-1315-4d86-988f-ab7fafafa831-maas-api-openshift-ca" pod "7bdd3394-1315-4d86-988f-ab7fafafa831" (UID: "7bdd3394-1315-4d86-988f-ab7fafafa831"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:24.310213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:24.310188 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bdd3394-1315-4d86-988f-ab7fafafa831-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:24.311537 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:24.311512 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bdd3394-1315-4d86-988f-ab7fafafa831-maas-api-xks-ca" pod "7bdd3394-1315-4d86-988f-ab7fafafa831" (UID: "7bdd3394-1315-4d86-988f-ab7fafafa831"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:06:24.311935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:24.311914 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdd3394-1315-4d86-988f-ab7fafafa831-kube-api-access-gwl4n" pod "7bdd3394-1315-4d86-988f-ab7fafafa831" (UID: "7bdd3394-1315-4d86-988f-ab7fafafa831"). InnerVolumeSpecName "kube-api-access-gwl4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:24.367055 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:24.367022 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:24.414085 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:24.414047 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bdd3394-1315-4d86-988f-ab7fafafa831-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:24.414085 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:24.414085 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwl4n\" (UniqueName: \"kubernetes.io/projected/7bdd3394-1315-4d86-988f-ab7fafafa831-kube-api-access-gwl4n\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:24.986077 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:24.986042 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p" event={"ID":"7bdd3394-1315-4d86-988f-ab7fafafa831","Type":"ContainerDied","Data":"fb01b828bcf648b5fee72a6baaa11a8ba59f119b398d763845f271be9aa12f31"} Oct 30 02:06:24.986269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:24.986080 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p" Oct 30 02:06:24.986269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:24.986131 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb01b828bcf648b5fee72a6baaa11a8ba59f119b398d763845f271be9aa12f31" Oct 30 02:06:25.221783 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:06:25.221732 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:06:25.585400 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.585364 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44"] Oct 30 02:06:25.585902 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.585889 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c2f3c8d5-6721-4780-a6dc-9d66c4785930" containerName="revoke-keys" Oct 30 02:06:25.585952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.585904 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2f3c8d5-6721-4780-a6dc-9d66c4785930" containerName="revoke-keys" Oct 30 02:06:25.585952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.585944 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7bdd3394-1315-4d86-988f-ab7fafafa831" containerName="revoke-keys" Oct 30 02:06:25.585952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.585950 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bdd3394-1315-4d86-988f-ab7fafafa831" containerName="revoke-keys" Oct 30 02:06:25.586054 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.586033 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bdd3394-1315-4d86-988f-ab7fafafa831" containerName="revoke-keys" Oct 30 02:06:25.586054 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.586043 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2f3c8d5-6721-4780-a6dc-9d66c4785930" containerName="revoke-keys" Oct 30 02:06:25.589432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.589416 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44" Oct 30 02:06:25.613207 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.613176 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:06:25.613373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.613257 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:06:25.613373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.613262 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:06:25.613373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.613262 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:06:25.617169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.617146 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44"] Oct 30 02:06:25.730400 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.730358 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/452f831c-f1d8-49df-900a-78f9a8972d98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-324255df-cnd44\" (UID: \"452f831c-f1d8-49df-900a-78f9a8972d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44" Oct 30 02:06:25.730588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.730422 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9crqn\" (UniqueName: \"kubernetes.io/projected/452f831c-f1d8-49df-900a-78f9a8972d98-kube-api-access-9crqn\") pod \"maas-api-revoke-sub-324255df-cnd44\" (UID: \"452f831c-f1d8-49df-900a-78f9a8972d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44" Oct 30 02:06:25.730640 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.730613 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/452f831c-f1d8-49df-900a-78f9a8972d98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-324255df-cnd44\" (UID: \"452f831c-f1d8-49df-900a-78f9a8972d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44" Oct 30 02:06:25.831758 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.831716 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/452f831c-f1d8-49df-900a-78f9a8972d98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-324255df-cnd44\" (UID: \"452f831c-f1d8-49df-900a-78f9a8972d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44" Oct 30 02:06:25.831970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.831778 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9crqn\" (UniqueName: \"kubernetes.io/projected/452f831c-f1d8-49df-900a-78f9a8972d98-kube-api-access-9crqn\") pod \"maas-api-revoke-sub-324255df-cnd44\" (UID: \"452f831c-f1d8-49df-900a-78f9a8972d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44" Oct 30 02:06:25.831970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.831822 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/452f831c-f1d8-49df-900a-78f9a8972d98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-324255df-cnd44\" (UID: \"452f831c-f1d8-49df-900a-78f9a8972d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44" Oct 30 02:06:25.832646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.832611 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/452f831c-f1d8-49df-900a-78f9a8972d98-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-324255df-cnd44\" (UID: \"452f831c-f1d8-49df-900a-78f9a8972d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44" Oct 30 02:06:25.834326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.834298 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/452f831c-f1d8-49df-900a-78f9a8972d98-maas-api-xks-ca\") pod \"maas-api-revoke-sub-324255df-cnd44\" (UID: \"452f831c-f1d8-49df-900a-78f9a8972d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44" Oct 30 02:06:25.853480 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.853423 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9crqn\" (UniqueName: \"kubernetes.io/projected/452f831c-f1d8-49df-900a-78f9a8972d98-kube-api-access-9crqn\") pod \"maas-api-revoke-sub-324255df-cnd44\" (UID: \"452f831c-f1d8-49df-900a-78f9a8972d98\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44" Oct 30 02:06:25.903228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.903197 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44" Oct 30 02:06:25.965713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:25.965681 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:26.211432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:26.211035 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44"] Oct 30 02:06:26.213727 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:06:26.213696 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452f831c_f1d8_49df_900a_78f9a8972d98.slice/crio-7941ddc607c632b0da97a5aa83bd4eea10d67881269d5e488bcb5e9832a9b3e8 WatchSource:0}: Error finding container 7941ddc607c632b0da97a5aa83bd4eea10d67881269d5e488bcb5e9832a9b3e8: Status 404 returned error can't find the container with id 7941ddc607c632b0da97a5aa83bd4eea10d67881269d5e488bcb5e9832a9b3e8 Oct 30 02:06:27.005022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:27.004980 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44" event={"ID":"452f831c-f1d8-49df-900a-78f9a8972d98","Type":"ContainerStarted","Data":"d6d654aa136d66e7c35fdab92405ccac9c57e9ff7d75d350ee41e4b9ca93cfcd"} Oct 30 02:06:27.005561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:27.005048 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44" event={"ID":"452f831c-f1d8-49df-900a-78f9a8972d98","Type":"ContainerStarted","Data":"7941ddc607c632b0da97a5aa83bd4eea10d67881269d5e488bcb5e9832a9b3e8"} Oct 30 02:06:27.055797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:27.055734 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44" podStartSLOduration=2.055720077 podStartE2EDuration="2.055720077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:27.045407244 +0000 UTC m=+2216.624198324" watchObservedRunningTime="2026-10-30 02:06:27.055720077 +0000 UTC m=+2216.634511159" Oct 30 02:06:27.573653 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:27.573621 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:28.014285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:28.014243 5998 generic.go:356] "Generic (PLEG): container finished" podID="452f831c-f1d8-49df-900a-78f9a8972d98" containerID="d6d654aa136d66e7c35fdab92405ccac9c57e9ff7d75d350ee41e4b9ca93cfcd" exitCode=0 Oct 30 02:06:28.014814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:28.014317 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44" event={"ID":"452f831c-f1d8-49df-900a-78f9a8972d98","Type":"ContainerDied","Data":"d6d654aa136d66e7c35fdab92405ccac9c57e9ff7d75d350ee41e4b9ca93cfcd"} Oct 30 02:06:29.173262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:29.173226 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44" Oct 30 02:06:29.181545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:29.181515 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:29.277399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:29.277189 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/452f831c-f1d8-49df-900a-78f9a8972d98-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/452f831c-f1d8-49df-900a-78f9a8972d98-maas-api-xks-ca\") pod \"452f831c-f1d8-49df-900a-78f9a8972d98\" (UID: \"452f831c-f1d8-49df-900a-78f9a8972d98\") " Oct 30 02:06:29.277755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:29.277724 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/452f831c-f1d8-49df-900a-78f9a8972d98-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/452f831c-f1d8-49df-900a-78f9a8972d98-maas-api-openshift-ca\") pod \"452f831c-f1d8-49df-900a-78f9a8972d98\" (UID: \"452f831c-f1d8-49df-900a-78f9a8972d98\") " Oct 30 02:06:29.278024 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:29.277984 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/452f831c-f1d8-49df-900a-78f9a8972d98-kube-api-access-9crqn\" (UniqueName: \"kubernetes.io/projected/452f831c-f1d8-49df-900a-78f9a8972d98-kube-api-access-9crqn\") pod \"452f831c-f1d8-49df-900a-78f9a8972d98\" (UID: \"452f831c-f1d8-49df-900a-78f9a8972d98\") " Oct 30 02:06:29.278442 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:29.278415 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/452f831c-f1d8-49df-900a-78f9a8972d98-maas-api-openshift-ca" pod "452f831c-f1d8-49df-900a-78f9a8972d98" (UID: "452f831c-f1d8-49df-900a-78f9a8972d98"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:29.278614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:29.278592 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/452f831c-f1d8-49df-900a-78f9a8972d98-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:29.280778 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:29.280750 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/452f831c-f1d8-49df-900a-78f9a8972d98-maas-api-xks-ca" pod "452f831c-f1d8-49df-900a-78f9a8972d98" (UID: "452f831c-f1d8-49df-900a-78f9a8972d98"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:06:29.281108 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:29.281066 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452f831c-f1d8-49df-900a-78f9a8972d98-kube-api-access-9crqn" pod "452f831c-f1d8-49df-900a-78f9a8972d98" (UID: "452f831c-f1d8-49df-900a-78f9a8972d98"). InnerVolumeSpecName "kube-api-access-9crqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:29.379464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:29.379428 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9crqn\" (UniqueName: \"kubernetes.io/projected/452f831c-f1d8-49df-900a-78f9a8972d98-kube-api-access-9crqn\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:29.379464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:29.379460 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/452f831c-f1d8-49df-900a-78f9a8972d98-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:30.029363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:30.029332 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44" Oct 30 02:06:30.029557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:30.029372 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44" event={"ID":"452f831c-f1d8-49df-900a-78f9a8972d98","Type":"ContainerDied","Data":"7941ddc607c632b0da97a5aa83bd4eea10d67881269d5e488bcb5e9832a9b3e8"} Oct 30 02:06:30.029557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:30.029431 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7941ddc607c632b0da97a5aa83bd4eea10d67881269d5e488bcb5e9832a9b3e8" Oct 30 02:06:30.925075 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:30.925031 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:32.595615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:32.595580 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:33.011688 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.011653 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s"] Oct 30 02:06:33.012534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.012509 5998 cpu_manager.go:414] "containerMap: removing container" podUID="452f831c-f1d8-49df-900a-78f9a8972d98" containerName="revoke-keys" Oct 30 02:06:33.012600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.012536 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="452f831c-f1d8-49df-900a-78f9a8972d98" containerName="revoke-keys" Oct 30 02:06:33.012682 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.012669 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="452f831c-f1d8-49df-900a-78f9a8972d98" containerName="revoke-keys" Oct 30 02:06:33.018338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.018312 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s" Oct 30 02:06:33.042283 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.042238 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:06:33.042530 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.042508 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:06:33.042674 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.042661 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:06:33.058336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.043574 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:06:33.061628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.061595 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s"] Oct 30 02:06:33.161462 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.161421 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mp6b\" (UniqueName: \"kubernetes.io/projected/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-kube-api-access-5mp6b\") pod \"maas-api-revoke-sub-d5faf1a6-xmd9s\" (UID: \"4ba37437-81c8-4edf-97e2-f16aadb9eaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s" Oct 30 02:06:33.161646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.161482 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5faf1a6-xmd9s\" (UID: \"4ba37437-81c8-4edf-97e2-f16aadb9eaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s" Oct 30 02:06:33.161754 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.161706 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5faf1a6-xmd9s\" (UID: \"4ba37437-81c8-4edf-97e2-f16aadb9eaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s" Oct 30 02:06:33.262817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.262721 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mp6b\" (UniqueName: \"kubernetes.io/projected/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-kube-api-access-5mp6b\") pod \"maas-api-revoke-sub-d5faf1a6-xmd9s\" (UID: \"4ba37437-81c8-4edf-97e2-f16aadb9eaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s" Oct 30 02:06:33.262817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.262777 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5faf1a6-xmd9s\" (UID: \"4ba37437-81c8-4edf-97e2-f16aadb9eaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s" Oct 30 02:06:33.263054 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.262914 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5faf1a6-xmd9s\" (UID: \"4ba37437-81c8-4edf-97e2-f16aadb9eaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s" Oct 30 02:06:33.263487 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.263462 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5faf1a6-xmd9s\" (UID: \"4ba37437-81c8-4edf-97e2-f16aadb9eaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s" Oct 30 02:06:33.265332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.265306 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5faf1a6-xmd9s\" (UID: \"4ba37437-81c8-4edf-97e2-f16aadb9eaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s" Oct 30 02:06:33.284953 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.284915 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mp6b\" (UniqueName: \"kubernetes.io/projected/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-kube-api-access-5mp6b\") pod \"maas-api-revoke-sub-d5faf1a6-xmd9s\" (UID: \"4ba37437-81c8-4edf-97e2-f16aadb9eaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s" Oct 30 02:06:33.336098 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.336053 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s" Oct 30 02:06:33.453842 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.453805 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7"] Oct 30 02:06:33.458858 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.458838 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7" Oct 30 02:06:33.476890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.476861 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7"] Oct 30 02:06:33.568085 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.567988 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9886070-5efc-43d6-a9bd-5a30014458ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7957d8ce-qwqq7\" (UID: \"c9886070-5efc-43d6-a9bd-5a30014458ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7" Oct 30 02:06:33.568085 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.568032 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9886070-5efc-43d6-a9bd-5a30014458ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7957d8ce-qwqq7\" (UID: \"c9886070-5efc-43d6-a9bd-5a30014458ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7" Oct 30 02:06:33.568320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.568232 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfj8k\" (UniqueName: \"kubernetes.io/projected/c9886070-5efc-43d6-a9bd-5a30014458ec-kube-api-access-cfj8k\") pod \"maas-api-revoke-sub-7957d8ce-qwqq7\" (UID: \"c9886070-5efc-43d6-a9bd-5a30014458ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7" Oct 30 02:06:33.640340 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.640313 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s"] Oct 30 02:06:33.643782 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:06:33.643740 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba37437_81c8_4edf_97e2_f16aadb9eaa6.slice/crio-08d3e7ef473d89804f797c1dd4aa21218137b61e45fa0b603759a6431cd7014c WatchSource:0}: Error finding container 08d3e7ef473d89804f797c1dd4aa21218137b61e45fa0b603759a6431cd7014c: Status 404 returned error can't find the container with id 08d3e7ef473d89804f797c1dd4aa21218137b61e45fa0b603759a6431cd7014c Oct 30 02:06:33.669438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.669408 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9886070-5efc-43d6-a9bd-5a30014458ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7957d8ce-qwqq7\" (UID: \"c9886070-5efc-43d6-a9bd-5a30014458ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7" Oct 30 02:06:33.669550 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.669454 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9886070-5efc-43d6-a9bd-5a30014458ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7957d8ce-qwqq7\" (UID: \"c9886070-5efc-43d6-a9bd-5a30014458ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7" Oct 30 02:06:33.669550 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.669532 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfj8k\" (UniqueName: \"kubernetes.io/projected/c9886070-5efc-43d6-a9bd-5a30014458ec-kube-api-access-cfj8k\") pod \"maas-api-revoke-sub-7957d8ce-qwqq7\" (UID: \"c9886070-5efc-43d6-a9bd-5a30014458ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7" Oct 30 02:06:33.670119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.670082 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9886070-5efc-43d6-a9bd-5a30014458ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7957d8ce-qwqq7\" (UID: \"c9886070-5efc-43d6-a9bd-5a30014458ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7" Oct 30 02:06:33.671637 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.671613 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9886070-5efc-43d6-a9bd-5a30014458ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7957d8ce-qwqq7\" (UID: \"c9886070-5efc-43d6-a9bd-5a30014458ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7" Oct 30 02:06:33.691024 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.691000 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfj8k\" (UniqueName: \"kubernetes.io/projected/c9886070-5efc-43d6-a9bd-5a30014458ec-kube-api-access-cfj8k\") pod \"maas-api-revoke-sub-7957d8ce-qwqq7\" (UID: \"c9886070-5efc-43d6-a9bd-5a30014458ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7" Oct 30 02:06:33.774112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:33.774068 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7" Oct 30 02:06:34.075241 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:34.075203 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s" event={"ID":"4ba37437-81c8-4edf-97e2-f16aadb9eaa6","Type":"ContainerStarted","Data":"366db7a9733d7971bba78b9b21bcd37984922e3833bdb0a89f2e2c4e40b524dd"} Oct 30 02:06:34.075419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:34.075269 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s" event={"ID":"4ba37437-81c8-4edf-97e2-f16aadb9eaa6","Type":"ContainerStarted","Data":"08d3e7ef473d89804f797c1dd4aa21218137b61e45fa0b603759a6431cd7014c"} Oct 30 02:06:34.093027 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:34.092999 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7"] Oct 30 02:06:34.096386 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:06:34.096309 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9886070_5efc_43d6_a9bd_5a30014458ec.slice/crio-a34aaab66cb7fdc5fa2fa7bfbc7b1e40859fd20f1a9902037641f7d2533ec2bc WatchSource:0}: Error finding container a34aaab66cb7fdc5fa2fa7bfbc7b1e40859fd20f1a9902037641f7d2533ec2bc: Status 404 returned error can't find the container with id a34aaab66cb7fdc5fa2fa7bfbc7b1e40859fd20f1a9902037641f7d2533ec2bc Oct 30 02:06:34.116322 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:34.115574 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s" podStartSLOduration=2.11555636 podStartE2EDuration="2.11555636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:34.11330256 +0000 UTC m=+2223.692093642" watchObservedRunningTime="2026-10-30 02:06:34.11555636 +0000 UTC m=+2223.694347443" Oct 30 02:06:34.295443 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:34.295409 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:34.829855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:34.829765 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:35.084235 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:35.084134 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7" event={"ID":"c9886070-5efc-43d6-a9bd-5a30014458ec","Type":"ContainerStarted","Data":"158aa69ded596f4e63ca223331161a0ebe4c338b4b6359dad700fefa77918526"} Oct 30 02:06:35.084235 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:35.084204 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7" event={"ID":"c9886070-5efc-43d6-a9bd-5a30014458ec","Type":"ContainerStarted","Data":"a34aaab66cb7fdc5fa2fa7bfbc7b1e40859fd20f1a9902037641f7d2533ec2bc"} Oct 30 02:06:35.086589 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:35.086556 5998 generic.go:356] "Generic (PLEG): container finished" podID="4ba37437-81c8-4edf-97e2-f16aadb9eaa6" containerID="366db7a9733d7971bba78b9b21bcd37984922e3833bdb0a89f2e2c4e40b524dd" exitCode=0 Oct 30 02:06:35.086726 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:35.086599 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s" event={"ID":"4ba37437-81c8-4edf-97e2-f16aadb9eaa6","Type":"ContainerDied","Data":"366db7a9733d7971bba78b9b21bcd37984922e3833bdb0a89f2e2c4e40b524dd"} Oct 30 02:06:35.129317 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:35.129256 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7" podStartSLOduration=2.129236879 podStartE2EDuration="2.129236879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:35.123245017 +0000 UTC m=+2224.702036112" watchObservedRunningTime="2026-10-30 02:06:35.129236879 +0000 UTC m=+2224.708027960" Oct 30 02:06:36.031143 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:36.031109 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:36.096980 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:36.096939 5998 generic.go:356] "Generic (PLEG): container finished" podID="c9886070-5efc-43d6-a9bd-5a30014458ec" containerID="158aa69ded596f4e63ca223331161a0ebe4c338b4b6359dad700fefa77918526" exitCode=0 Oct 30 02:06:36.097184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:36.097013 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7" event={"ID":"c9886070-5efc-43d6-a9bd-5a30014458ec","Type":"ContainerDied","Data":"158aa69ded596f4e63ca223331161a0ebe4c338b4b6359dad700fefa77918526"} Oct 30 02:06:36.163134 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:36.158503 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:36.273573 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:36.273533 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:36.329970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:36.329937 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s" Oct 30 02:06:36.406350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:36.406306 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-maas-api-xks-ca\") pod \"4ba37437-81c8-4edf-97e2-f16aadb9eaa6\" (UID: \"4ba37437-81c8-4edf-97e2-f16aadb9eaa6\") " Oct 30 02:06:36.406546 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:36.406424 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-kube-api-access-5mp6b\" (UniqueName: \"kubernetes.io/projected/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-kube-api-access-5mp6b\") pod \"4ba37437-81c8-4edf-97e2-f16aadb9eaa6\" (UID: \"4ba37437-81c8-4edf-97e2-f16aadb9eaa6\") " Oct 30 02:06:36.406889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:36.406831 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-maas-api-openshift-ca\") pod \"4ba37437-81c8-4edf-97e2-f16aadb9eaa6\" (UID: \"4ba37437-81c8-4edf-97e2-f16aadb9eaa6\") " Oct 30 02:06:36.407144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:36.407116 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-maas-api-openshift-ca" pod "4ba37437-81c8-4edf-97e2-f16aadb9eaa6" (UID: "4ba37437-81c8-4edf-97e2-f16aadb9eaa6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:36.407517 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:36.407491 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:36.410538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:36.410513 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-maas-api-xks-ca" pod "4ba37437-81c8-4edf-97e2-f16aadb9eaa6" (UID: "4ba37437-81c8-4edf-97e2-f16aadb9eaa6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:06:36.410816 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:36.410791 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-kube-api-access-5mp6b" pod "4ba37437-81c8-4edf-97e2-f16aadb9eaa6" (UID: "4ba37437-81c8-4edf-97e2-f16aadb9eaa6"). InnerVolumeSpecName "kube-api-access-5mp6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:36.509109 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:36.509055 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:36.509469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:36.509448 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mp6b\" (UniqueName: \"kubernetes.io/projected/4ba37437-81c8-4edf-97e2-f16aadb9eaa6-kube-api-access-5mp6b\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:36.541445 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:36.541408 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:36.651112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:36.650562 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:37.106512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:37.106479 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s" Oct 30 02:06:37.106974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:37.106504 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s" event={"ID":"4ba37437-81c8-4edf-97e2-f16aadb9eaa6","Type":"ContainerDied","Data":"08d3e7ef473d89804f797c1dd4aa21218137b61e45fa0b603759a6431cd7014c"} Oct 30 02:06:37.106974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:37.106568 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08d3e7ef473d89804f797c1dd4aa21218137b61e45fa0b603759a6431cd7014c" Oct 30 02:06:37.380167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:37.375856 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:06:37.443312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:37.443281 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7" Oct 30 02:06:37.530417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:37.530383 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9886070-5efc-43d6-a9bd-5a30014458ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9886070-5efc-43d6-a9bd-5a30014458ec-maas-api-xks-ca\") pod \"c9886070-5efc-43d6-a9bd-5a30014458ec\" (UID: \"c9886070-5efc-43d6-a9bd-5a30014458ec\") " Oct 30 02:06:37.530770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:37.530745 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9886070-5efc-43d6-a9bd-5a30014458ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9886070-5efc-43d6-a9bd-5a30014458ec-maas-api-openshift-ca\") pod \"c9886070-5efc-43d6-a9bd-5a30014458ec\" (UID: \"c9886070-5efc-43d6-a9bd-5a30014458ec\") " Oct 30 02:06:37.531233 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:37.531211 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9886070-5efc-43d6-a9bd-5a30014458ec-kube-api-access-cfj8k\" (UniqueName: \"kubernetes.io/projected/c9886070-5efc-43d6-a9bd-5a30014458ec-kube-api-access-cfj8k\") pod \"c9886070-5efc-43d6-a9bd-5a30014458ec\" (UID: \"c9886070-5efc-43d6-a9bd-5a30014458ec\") " Oct 30 02:06:37.540302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:37.540272 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9886070-5efc-43d6-a9bd-5a30014458ec-maas-api-openshift-ca" pod "c9886070-5efc-43d6-a9bd-5a30014458ec" (UID: "c9886070-5efc-43d6-a9bd-5a30014458ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:37.543170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:37.543148 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9886070-5efc-43d6-a9bd-5a30014458ec-maas-api-xks-ca" pod "c9886070-5efc-43d6-a9bd-5a30014458ec" (UID: "c9886070-5efc-43d6-a9bd-5a30014458ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:06:37.545519 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:37.545488 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9886070-5efc-43d6-a9bd-5a30014458ec-kube-api-access-cfj8k" pod "c9886070-5efc-43d6-a9bd-5a30014458ec" (UID: "c9886070-5efc-43d6-a9bd-5a30014458ec"). InnerVolumeSpecName "kube-api-access-cfj8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:37.633815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:37.633711 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfj8k\" (UniqueName: \"kubernetes.io/projected/c9886070-5efc-43d6-a9bd-5a30014458ec-kube-api-access-cfj8k\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:37.633815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:37.633753 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9886070-5efc-43d6-a9bd-5a30014458ec-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:37.633815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:37.633768 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9886070-5efc-43d6-a9bd-5a30014458ec-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:38.113111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:38.107045 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:38.129115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:38.125490 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7" event={"ID":"c9886070-5efc-43d6-a9bd-5a30014458ec","Type":"ContainerDied","Data":"a34aaab66cb7fdc5fa2fa7bfbc7b1e40859fd20f1a9902037641f7d2533ec2bc"} Oct 30 02:06:38.129115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:38.125551 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a34aaab66cb7fdc5fa2fa7bfbc7b1e40859fd20f1a9902037641f7d2533ec2bc" Oct 30 02:06:38.129115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:38.125620 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7" Oct 30 02:06:40.917293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:40.917232 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:06:40.918136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:40.918081 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" podUID="2b27c952-8035-4847-b243-4471250d3d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:41.079783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:41.079731 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:41.429486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:41.429428 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:41.541599 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:41.541567 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:41.818606 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:41.818483 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:43.420989 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:43.420956 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:43.571898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:43.571861 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:43.833646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:43.833552 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr"] Oct 30 02:06:43.841114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:43.836431 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c9886070-5efc-43d6-a9bd-5a30014458ec" containerName="revoke-keys" Oct 30 02:06:43.856304 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:43.856273 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9886070-5efc-43d6-a9bd-5a30014458ec" containerName="revoke-keys" Oct 30 02:06:43.856557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:43.856541 5998 cpu_manager.go:414] "containerMap: removing container" podUID="4ba37437-81c8-4edf-97e2-f16aadb9eaa6" containerName="revoke-keys" Oct 30 02:06:43.856642 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:43.856632 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba37437-81c8-4edf-97e2-f16aadb9eaa6" containerName="revoke-keys" Oct 30 02:06:43.856896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:43.856883 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9886070-5efc-43d6-a9bd-5a30014458ec" containerName="revoke-keys" Oct 30 02:06:43.856995 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:43.856987 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba37437-81c8-4edf-97e2-f16aadb9eaa6" containerName="revoke-keys" Oct 30 02:06:43.879335 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:43.879304 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr" Oct 30 02:06:43.886286 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:43.886243 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr"] Oct 30 02:06:43.896592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:43.896558 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:06:43.907463 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:43.907359 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:06:43.938516 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:43.938477 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57jbx\" (UniqueName: \"kubernetes.io/projected/96f5ee14-8359-4b68-83d5-8f71edca96ef-kube-api-access-57jbx\") pod \"maas-api-revoke-sub-e147c024-cwrpr\" (UID: \"96f5ee14-8359-4b68-83d5-8f71edca96ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr" Oct 30 02:06:43.938821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:43.938800 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96f5ee14-8359-4b68-83d5-8f71edca96ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e147c024-cwrpr\" (UID: \"96f5ee14-8359-4b68-83d5-8f71edca96ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr" Oct 30 02:06:43.939050 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:43.939035 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96f5ee14-8359-4b68-83d5-8f71edca96ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e147c024-cwrpr\" (UID: \"96f5ee14-8359-4b68-83d5-8f71edca96ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr" Oct 30 02:06:43.945815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:43.945779 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:06:43.945972 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:43.945937 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:06:44.040283 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:44.040235 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96f5ee14-8359-4b68-83d5-8f71edca96ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e147c024-cwrpr\" (UID: \"96f5ee14-8359-4b68-83d5-8f71edca96ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr" Oct 30 02:06:44.040497 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:44.040406 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96f5ee14-8359-4b68-83d5-8f71edca96ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e147c024-cwrpr\" (UID: \"96f5ee14-8359-4b68-83d5-8f71edca96ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr" Oct 30 02:06:44.040573 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:44.040541 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57jbx\" (UniqueName: \"kubernetes.io/projected/96f5ee14-8359-4b68-83d5-8f71edca96ef-kube-api-access-57jbx\") pod \"maas-api-revoke-sub-e147c024-cwrpr\" (UID: \"96f5ee14-8359-4b68-83d5-8f71edca96ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr" Oct 30 02:06:44.041147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:44.041075 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96f5ee14-8359-4b68-83d5-8f71edca96ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e147c024-cwrpr\" (UID: \"96f5ee14-8359-4b68-83d5-8f71edca96ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr" Oct 30 02:06:44.043165 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:44.043135 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96f5ee14-8359-4b68-83d5-8f71edca96ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e147c024-cwrpr\" (UID: \"96f5ee14-8359-4b68-83d5-8f71edca96ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr" Oct 30 02:06:44.062490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:44.062463 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57jbx\" (UniqueName: \"kubernetes.io/projected/96f5ee14-8359-4b68-83d5-8f71edca96ef-kube-api-access-57jbx\") pod \"maas-api-revoke-sub-e147c024-cwrpr\" (UID: \"96f5ee14-8359-4b68-83d5-8f71edca96ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr" Oct 30 02:06:44.246834 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:44.246800 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr" Oct 30 02:06:44.399517 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:44.399481 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:44.554669 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:44.554643 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr"] Oct 30 02:06:44.557608 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:06:44.557578 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f5ee14_8359_4b68_83d5_8f71edca96ef.slice/crio-0676b027003f77e9e3f3b842fa593c0482e21ff7a17b478886e3543611ae5969 WatchSource:0}: Error finding container 0676b027003f77e9e3f3b842fa593c0482e21ff7a17b478886e3543611ae5969: Status 404 returned error can't find the container with id 0676b027003f77e9e3f3b842fa593c0482e21ff7a17b478886e3543611ae5969 Oct 30 02:06:45.230101 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:45.230050 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr" event={"ID":"96f5ee14-8359-4b68-83d5-8f71edca96ef","Type":"ContainerStarted","Data":"d2303845278c70cf593f93ec37ff12959676a62555f1e117f0fd352245652911"} Oct 30 02:06:45.230294 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:45.230147 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr" event={"ID":"96f5ee14-8359-4b68-83d5-8f71edca96ef","Type":"ContainerStarted","Data":"0676b027003f77e9e3f3b842fa593c0482e21ff7a17b478886e3543611ae5969"} Oct 30 02:06:45.271335 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:45.271278 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr" podStartSLOduration=2.271261136 podStartE2EDuration="2.271261136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:45.267053009 +0000 UTC m=+2234.845844091" watchObservedRunningTime="2026-10-30 02:06:45.271261136 +0000 UTC m=+2234.850052216" Oct 30 02:06:46.241109 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:46.240911 5998 generic.go:356] "Generic (PLEG): container finished" podID="96f5ee14-8359-4b68-83d5-8f71edca96ef" containerID="d2303845278c70cf593f93ec37ff12959676a62555f1e117f0fd352245652911" exitCode=0 Oct 30 02:06:46.241109 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:46.241004 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr" event={"ID":"96f5ee14-8359-4b68-83d5-8f71edca96ef","Type":"ContainerDied","Data":"d2303845278c70cf593f93ec37ff12959676a62555f1e117f0fd352245652911"} Oct 30 02:06:46.301268 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:46.301228 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:47.282006 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.281969 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:47.407873 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.407848 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr" Oct 30 02:06:47.491246 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.491206 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96f5ee14-8359-4b68-83d5-8f71edca96ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96f5ee14-8359-4b68-83d5-8f71edca96ef-maas-api-xks-ca\") pod \"96f5ee14-8359-4b68-83d5-8f71edca96ef\" (UID: \"96f5ee14-8359-4b68-83d5-8f71edca96ef\") " Oct 30 02:06:47.491900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.491875 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96f5ee14-8359-4b68-83d5-8f71edca96ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96f5ee14-8359-4b68-83d5-8f71edca96ef-maas-api-openshift-ca\") pod \"96f5ee14-8359-4b68-83d5-8f71edca96ef\" (UID: \"96f5ee14-8359-4b68-83d5-8f71edca96ef\") " Oct 30 02:06:47.492569 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.492553 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96f5ee14-8359-4b68-83d5-8f71edca96ef-kube-api-access-57jbx\" (UniqueName: \"kubernetes.io/projected/96f5ee14-8359-4b68-83d5-8f71edca96ef-kube-api-access-57jbx\") pod \"96f5ee14-8359-4b68-83d5-8f71edca96ef\" (UID: \"96f5ee14-8359-4b68-83d5-8f71edca96ef\") " Oct 30 02:06:47.493540 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.492412 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f5ee14-8359-4b68-83d5-8f71edca96ef-maas-api-openshift-ca" pod "96f5ee14-8359-4b68-83d5-8f71edca96ef" (UID: "96f5ee14-8359-4b68-83d5-8f71edca96ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:47.493971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.493933 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96f5ee14-8359-4b68-83d5-8f71edca96ef-maas-api-xks-ca" pod "96f5ee14-8359-4b68-83d5-8f71edca96ef" (UID: "96f5ee14-8359-4b68-83d5-8f71edca96ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:06:47.495976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.495940 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f5ee14-8359-4b68-83d5-8f71edca96ef-kube-api-access-57jbx" pod "96f5ee14-8359-4b68-83d5-8f71edca96ef" (UID: "96f5ee14-8359-4b68-83d5-8f71edca96ef"). InnerVolumeSpecName "kube-api-access-57jbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:47.594942 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.594907 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57jbx\" (UniqueName: \"kubernetes.io/projected/96f5ee14-8359-4b68-83d5-8f71edca96ef-kube-api-access-57jbx\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:47.594942 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.594937 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96f5ee14-8359-4b68-83d5-8f71edca96ef-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:47.594942 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.594947 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96f5ee14-8359-4b68-83d5-8f71edca96ef-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:47.708305 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.708268 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx"] Oct 30 02:06:47.708996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.708965 5998 cpu_manager.go:414] "containerMap: removing container" podUID="96f5ee14-8359-4b68-83d5-8f71edca96ef" containerName="revoke-keys" Oct 30 02:06:47.709152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.708996 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96f5ee14-8359-4b68-83d5-8f71edca96ef" containerName="revoke-keys" Oct 30 02:06:47.709225 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.709214 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="96f5ee14-8359-4b68-83d5-8f71edca96ef" containerName="revoke-keys" Oct 30 02:06:47.713011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.712984 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx" Oct 30 02:06:47.734180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.734149 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx"] Oct 30 02:06:47.797140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.797028 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42e10aff-lfhcx\" (UID: \"fee54a00-be9c-4ebd-bbd1-bdca4e49b70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx" Oct 30 02:06:47.797290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.797262 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42e10aff-lfhcx\" (UID: \"fee54a00-be9c-4ebd-bbd1-bdca4e49b70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx" Oct 30 02:06:47.797290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.797287 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgcvx\" (UniqueName: \"kubernetes.io/projected/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-kube-api-access-qgcvx\") pod \"maas-api-revoke-sub-42e10aff-lfhcx\" (UID: \"fee54a00-be9c-4ebd-bbd1-bdca4e49b70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx" Oct 30 02:06:47.882305 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.882270 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:47.898690 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.898653 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42e10aff-lfhcx\" (UID: \"fee54a00-be9c-4ebd-bbd1-bdca4e49b70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx" Oct 30 02:06:47.898938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.898918 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42e10aff-lfhcx\" (UID: \"fee54a00-be9c-4ebd-bbd1-bdca4e49b70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx" Oct 30 02:06:47.898994 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.898956 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgcvx\" (UniqueName: \"kubernetes.io/projected/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-kube-api-access-qgcvx\") pod \"maas-api-revoke-sub-42e10aff-lfhcx\" (UID: \"fee54a00-be9c-4ebd-bbd1-bdca4e49b70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx" Oct 30 02:06:47.899622 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.899599 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42e10aff-lfhcx\" (UID: \"fee54a00-be9c-4ebd-bbd1-bdca4e49b70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx" Oct 30 02:06:47.901131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.901073 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42e10aff-lfhcx\" (UID: \"fee54a00-be9c-4ebd-bbd1-bdca4e49b70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx" Oct 30 02:06:47.920770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:47.920736 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgcvx\" (UniqueName: \"kubernetes.io/projected/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-kube-api-access-qgcvx\") pod \"maas-api-revoke-sub-42e10aff-lfhcx\" (UID: \"fee54a00-be9c-4ebd-bbd1-bdca4e49b70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx" Oct 30 02:06:48.031461 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:48.031425 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx" Oct 30 02:06:48.259065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:48.259031 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr" event={"ID":"96f5ee14-8359-4b68-83d5-8f71edca96ef","Type":"ContainerDied","Data":"0676b027003f77e9e3f3b842fa593c0482e21ff7a17b478886e3543611ae5969"} Oct 30 02:06:48.259261 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:48.259068 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr" Oct 30 02:06:48.259261 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:48.259080 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0676b027003f77e9e3f3b842fa593c0482e21ff7a17b478886e3543611ae5969" Oct 30 02:06:48.341501 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:48.339558 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx"] Oct 30 02:06:48.343947 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:06:48.343906 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee54a00_be9c_4ebd_bbd1_bdca4e49b70d.slice/crio-5361256be4976619266484745b0c5d33e231d7d48e29d5c80a47fe12d659d8b3 WatchSource:0}: Error finding container 5361256be4976619266484745b0c5d33e231d7d48e29d5c80a47fe12d659d8b3: Status 404 returned error can't find the container with id 5361256be4976619266484745b0c5d33e231d7d48e29d5c80a47fe12d659d8b3 Oct 30 02:06:49.267325 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:49.267293 5998 generic.go:356] "Generic (PLEG): container finished" podID="fee54a00-be9c-4ebd-bbd1-bdca4e49b70d" containerID="d3e81082d2b84227187b2540ce57626a3304726658e1b8d73a4d49e9b1e03e51" exitCode=0 Oct 30 02:06:49.267503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:49.267373 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx" event={"ID":"fee54a00-be9c-4ebd-bbd1-bdca4e49b70d","Type":"ContainerDied","Data":"d3e81082d2b84227187b2540ce57626a3304726658e1b8d73a4d49e9b1e03e51"} Oct 30 02:06:49.267503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:49.267429 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx" event={"ID":"fee54a00-be9c-4ebd-bbd1-bdca4e49b70d","Type":"ContainerStarted","Data":"5361256be4976619266484745b0c5d33e231d7d48e29d5c80a47fe12d659d8b3"} Oct 30 02:06:49.451318 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:49.451287 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:50.436484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:50.436456 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx" Oct 30 02:06:50.536521 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:50.536421 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-maas-api-xks-ca\") pod \"fee54a00-be9c-4ebd-bbd1-bdca4e49b70d\" (UID: \"fee54a00-be9c-4ebd-bbd1-bdca4e49b70d\") " Oct 30 02:06:50.537002 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:50.536697 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-kube-api-access-qgcvx\" (UniqueName: \"kubernetes.io/projected/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-kube-api-access-qgcvx\") pod \"fee54a00-be9c-4ebd-bbd1-bdca4e49b70d\" (UID: \"fee54a00-be9c-4ebd-bbd1-bdca4e49b70d\") " Oct 30 02:06:50.537002 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:50.536736 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-maas-api-openshift-ca\") pod \"fee54a00-be9c-4ebd-bbd1-bdca4e49b70d\" (UID: \"fee54a00-be9c-4ebd-bbd1-bdca4e49b70d\") " Oct 30 02:06:50.537863 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:50.537831 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-maas-api-openshift-ca" pod "fee54a00-be9c-4ebd-bbd1-bdca4e49b70d" (UID: "fee54a00-be9c-4ebd-bbd1-bdca4e49b70d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:50.540197 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:50.540082 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-kube-api-access-qgcvx" pod "fee54a00-be9c-4ebd-bbd1-bdca4e49b70d" (UID: "fee54a00-be9c-4ebd-bbd1-bdca4e49b70d"). InnerVolumeSpecName "kube-api-access-qgcvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:50.540567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:50.540545 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-maas-api-xks-ca" pod "fee54a00-be9c-4ebd-bbd1-bdca4e49b70d" (UID: "fee54a00-be9c-4ebd-bbd1-bdca4e49b70d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:06:50.639770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:50.639720 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.639770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:50.639767 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgcvx\" (UniqueName: \"kubernetes.io/projected/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-kube-api-access-qgcvx\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.640031 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:50.639785 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.961362 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:50.961323 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:51.137656 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:51.137608 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:51.284221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:51.284134 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx" event={"ID":"fee54a00-be9c-4ebd-bbd1-bdca4e49b70d","Type":"ContainerDied","Data":"5361256be4976619266484745b0c5d33e231d7d48e29d5c80a47fe12d659d8b3"} Oct 30 02:06:51.284221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:51.284197 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5361256be4976619266484745b0c5d33e231d7d48e29d5c80a47fe12d659d8b3" Oct 30 02:06:51.284425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:51.284263 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx" Oct 30 02:06:51.455370 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:51.455335 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:51.577367 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:51.577281 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:52.376152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:52.376077 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:53.018287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:53.018253 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:54.390323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.390284 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j"] Oct 30 02:06:54.391787 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.391759 5998 cpu_manager.go:414] "containerMap: removing container" podUID="fee54a00-be9c-4ebd-bbd1-bdca4e49b70d" containerName="revoke-keys" Oct 30 02:06:54.391948 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.391932 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee54a00-be9c-4ebd-bbd1-bdca4e49b70d" containerName="revoke-keys" Oct 30 02:06:54.392215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.392198 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee54a00-be9c-4ebd-bbd1-bdca4e49b70d" containerName="revoke-keys" Oct 30 02:06:54.421022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.420987 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j" Oct 30 02:06:54.431234 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.431202 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j"] Oct 30 02:06:54.450170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.450078 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:06:54.466565 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.463432 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:06:54.466565 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.463585 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:06:54.466565 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.463666 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:06:54.526121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.512874 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e326b0d1-a861-4e96-b418-b9ad631f6a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac627e96-hbd7j\" (UID: \"e326b0d1-a861-4e96-b418-b9ad631f6a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j" Oct 30 02:06:54.526575 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.526539 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e326b0d1-a861-4e96-b418-b9ad631f6a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac627e96-hbd7j\" (UID: \"e326b0d1-a861-4e96-b418-b9ad631f6a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j" Oct 30 02:06:54.535119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.526796 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd5qz\" (UniqueName: \"kubernetes.io/projected/e326b0d1-a861-4e96-b418-b9ad631f6a33-kube-api-access-gd5qz\") pod \"maas-api-revoke-sub-ac627e96-hbd7j\" (UID: \"e326b0d1-a861-4e96-b418-b9ad631f6a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j" Oct 30 02:06:54.669177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.654166 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e326b0d1-a861-4e96-b418-b9ad631f6a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac627e96-hbd7j\" (UID: \"e326b0d1-a861-4e96-b418-b9ad631f6a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j" Oct 30 02:06:54.669609 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.669576 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd5qz\" (UniqueName: \"kubernetes.io/projected/e326b0d1-a861-4e96-b418-b9ad631f6a33-kube-api-access-gd5qz\") pod \"maas-api-revoke-sub-ac627e96-hbd7j\" (UID: \"e326b0d1-a861-4e96-b418-b9ad631f6a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j" Oct 30 02:06:54.670083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.670045 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e326b0d1-a861-4e96-b418-b9ad631f6a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac627e96-hbd7j\" (UID: \"e326b0d1-a861-4e96-b418-b9ad631f6a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j" Oct 30 02:06:54.703747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.703708 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e326b0d1-a861-4e96-b418-b9ad631f6a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac627e96-hbd7j\" (UID: \"e326b0d1-a861-4e96-b418-b9ad631f6a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j" Oct 30 02:06:54.712388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.712335 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e326b0d1-a861-4e96-b418-b9ad631f6a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac627e96-hbd7j\" (UID: \"e326b0d1-a861-4e96-b418-b9ad631f6a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j" Oct 30 02:06:54.836114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:54.832935 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd5qz\" (UniqueName: \"kubernetes.io/projected/e326b0d1-a861-4e96-b418-b9ad631f6a33-kube-api-access-gd5qz\") pod \"maas-api-revoke-sub-ac627e96-hbd7j\" (UID: \"e326b0d1-a861-4e96-b418-b9ad631f6a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j" Oct 30 02:06:55.129413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:55.129318 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j" Oct 30 02:06:55.479116 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:06:55.466462 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:06:55.724943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:55.724718 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:55.907098 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:55.906979 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:55.959827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:55.958293 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:06:55.959827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:55.958363 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" podUID="2b27c952-8035-4847-b243-4471250d3d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:06:56.063662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:56.063624 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:56.093498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:56.093471 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j"] Oct 30 02:06:56.355162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:56.355125 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j" event={"ID":"e326b0d1-a861-4e96-b418-b9ad631f6a33","Type":"ContainerStarted","Data":"095a336a383ef3eb8e9aca72cb47b62f674c2783b909b41bceccb9f087963ef4"} Oct 30 02:06:56.466123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:56.466018 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:56.954843 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:56.954807 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:57.365601 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:57.365562 5998 generic.go:356] "Generic (PLEG): container finished" podID="e326b0d1-a861-4e96-b418-b9ad631f6a33" containerID="d7841df9c853741049a9a34f4c31d59dbcf8d01b191ebae3fed29a090371731b" exitCode=0 Oct 30 02:06:57.365775 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:57.365611 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j" event={"ID":"e326b0d1-a861-4e96-b418-b9ad631f6a33","Type":"ContainerDied","Data":"d7841df9c853741049a9a34f4c31d59dbcf8d01b191ebae3fed29a090371731b"} Oct 30 02:06:58.025375 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:58.025330 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:58.138573 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:58.138532 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:58.661028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:58.660987 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j" Oct 30 02:06:58.675151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:58.675123 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e326b0d1-a861-4e96-b418-b9ad631f6a33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e326b0d1-a861-4e96-b418-b9ad631f6a33-maas-api-xks-ca\") pod \"e326b0d1-a861-4e96-b418-b9ad631f6a33\" (UID: \"e326b0d1-a861-4e96-b418-b9ad631f6a33\") " Oct 30 02:06:58.675334 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:58.675221 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e326b0d1-a861-4e96-b418-b9ad631f6a33-kube-api-access-gd5qz\" (UniqueName: \"kubernetes.io/projected/e326b0d1-a861-4e96-b418-b9ad631f6a33-kube-api-access-gd5qz\") pod \"e326b0d1-a861-4e96-b418-b9ad631f6a33\" (UID: \"e326b0d1-a861-4e96-b418-b9ad631f6a33\") " Oct 30 02:06:58.675334 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:58.675276 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e326b0d1-a861-4e96-b418-b9ad631f6a33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e326b0d1-a861-4e96-b418-b9ad631f6a33-maas-api-openshift-ca\") pod \"e326b0d1-a861-4e96-b418-b9ad631f6a33\" (UID: \"e326b0d1-a861-4e96-b418-b9ad631f6a33\") " Oct 30 02:06:58.675738 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:58.675708 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e326b0d1-a861-4e96-b418-b9ad631f6a33-maas-api-openshift-ca" pod "e326b0d1-a861-4e96-b418-b9ad631f6a33" (UID: "e326b0d1-a861-4e96-b418-b9ad631f6a33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:58.675863 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:58.675845 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e326b0d1-a861-4e96-b418-b9ad631f6a33-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:58.677380 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:58.677352 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e326b0d1-a861-4e96-b418-b9ad631f6a33-maas-api-xks-ca" pod "e326b0d1-a861-4e96-b418-b9ad631f6a33" (UID: "e326b0d1-a861-4e96-b418-b9ad631f6a33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:06:58.677519 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:58.677493 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e326b0d1-a861-4e96-b418-b9ad631f6a33-kube-api-access-gd5qz" pod "e326b0d1-a861-4e96-b418-b9ad631f6a33" (UID: "e326b0d1-a861-4e96-b418-b9ad631f6a33"). InnerVolumeSpecName "kube-api-access-gd5qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:58.776943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:58.776835 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e326b0d1-a861-4e96-b418-b9ad631f6a33-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:58.776943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:58.776872 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd5qz\" (UniqueName: \"kubernetes.io/projected/e326b0d1-a861-4e96-b418-b9ad631f6a33-kube-api-access-gd5qz\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:06:59.152762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:59.152678 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:59.287781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:59.287746 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:59.482654 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:59.482623 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j" Oct 30 02:06:59.482848 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:59.482648 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j" event={"ID":"e326b0d1-a861-4e96-b418-b9ad631f6a33","Type":"ContainerDied","Data":"095a336a383ef3eb8e9aca72cb47b62f674c2783b909b41bceccb9f087963ef4"} Oct 30 02:06:59.482848 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:59.482733 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="095a336a383ef3eb8e9aca72cb47b62f674c2783b909b41bceccb9f087963ef4" Oct 30 02:06:59.665028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:59.664992 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:06:59.802169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:06:59.801938 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:01.137446 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:01.137409 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:01.244950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:01.244896 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:01.353983 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:01.353917 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:03.188342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:03.188298 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:03.808701 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:03.808668 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:03.933484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:03.933445 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:04.983511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:04.983472 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j"] Oct 30 02:07:04.984413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:04.984387 5998 cpu_manager.go:414] "containerMap: removing container" podUID="e326b0d1-a861-4e96-b418-b9ad631f6a33" containerName="revoke-keys" Oct 30 02:07:04.984413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:04.984414 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e326b0d1-a861-4e96-b418-b9ad631f6a33" containerName="revoke-keys" Oct 30 02:07:04.984587 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:04.984539 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="e326b0d1-a861-4e96-b418-b9ad631f6a33" containerName="revoke-keys" Oct 30 02:07:04.989273 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:04.989247 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.003602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.003576 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q"] Oct 30 02:07:05.010694 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.010672 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.013672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.013648 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j"] Oct 30 02:07:05.049260 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.049176 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q"] Oct 30 02:07:05.074785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.074750 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.074963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.074911 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.074963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.074947 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.075079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.074972 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.075079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.075057 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m72w5\" (UniqueName: \"kubernetes.io/projected/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-kube-api-access-m72w5\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.075214 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.075164 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.075214 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.075196 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.075322 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.075222 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.075322 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.075280 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.075414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.075318 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.075414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.075362 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.075498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.075437 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7src\" (UniqueName: \"kubernetes.io/projected/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-kube-api-access-f7src\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.075498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.075477 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.075573 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.075500 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.075573 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.075559 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.075643 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.075625 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.075692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.075648 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.075768 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.075756 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.176496 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.176461 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.176703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.176548 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.176703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.176571 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.176703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.176612 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.176703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.176646 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.176943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.176718 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.176943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.176748 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.176943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.176772 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.176943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.176798 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m72w5\" (UniqueName: \"kubernetes.io/projected/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-kube-api-access-m72w5\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.176943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.176860 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.176943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.176894 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.176943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.176934 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.177388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.176959 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.177388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.176983 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.177388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.176997 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.177388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.177010 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.177388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.177050 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.177388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.177126 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7src\" (UniqueName: \"kubernetes.io/projected/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-kube-api-access-f7src\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.177388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.177201 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.177388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.177233 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.177388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.177305 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.177864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.177536 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.177864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.177619 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-workload-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.177864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.177695 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istiod-ca-cert\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.177864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.177719 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.178084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.177924 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-credential-socket\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.178084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.177983 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-workload-certs\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.178306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.178279 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-data\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.179699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.179674 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.180147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.180125 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.180430 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.180410 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-envoy\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.180487 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.180471 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-podinfo\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.202206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.202168 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.202341 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.202170 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-token\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.202341 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.202318 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7src\" (UniqueName: \"kubernetes.io/projected/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-kube-api-access-f7src\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.207166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.207140 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m72w5\" (UniqueName: \"kubernetes.io/projected/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-kube-api-access-m72w5\") pod \"e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.225473 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.225448 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:05.308212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.308128 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:05.327189 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.327160 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:05.578560 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.578474 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:05.651738 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.651711 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j"] Oct 30 02:07:05.653597 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:07:05.653571 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd00e78e_d69e_4a6c_99a4_ba99c76cc5b3.slice/crio-d5aef54cd74d9b5e352acbb97473e52e01e4bd20df8c42e980e4613959112867 WatchSource:0}: Error finding container d5aef54cd74d9b5e352acbb97473e52e01e4bd20df8c42e980e4613959112867: Status 404 returned error can't find the container with id d5aef54cd74d9b5e352acbb97473e52e01e4bd20df8c42e980e4613959112867 Oct 30 02:07:05.656120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.656072 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:07:05.656221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.656190 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:07:05.656264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.656222 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:07:05.692491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.692455 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:05.697359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.697334 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q"] Oct 30 02:07:05.697705 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:07:05.697674 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf935c19c_9ed9_4f78_aa0a_d52823a3de7b.slice/crio-880d5d3111a76ebbfa593394824a8d7f624b484d99462acd489c1da4b8341651 WatchSource:0}: Error finding container 880d5d3111a76ebbfa593394824a8d7f624b484d99462acd489c1da4b8341651: Status 404 returned error can't find the container with id 880d5d3111a76ebbfa593394824a8d7f624b484d99462acd489c1da4b8341651 Oct 30 02:07:05.700478 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.700449 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:07:05.700588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.700525 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:07:05.700654 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:05.700583 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:07:06.576851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:06.576804 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" event={"ID":"f935c19c-9ed9-4f78-aa0a-d52823a3de7b","Type":"ContainerStarted","Data":"65e0034f3432e095a1dcda39a7952727afee1c00b78b680ff5d14b191dbba77b"} Oct 30 02:07:06.576851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:06.576858 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" event={"ID":"f935c19c-9ed9-4f78-aa0a-d52823a3de7b","Type":"ContainerStarted","Data":"880d5d3111a76ebbfa593394824a8d7f624b484d99462acd489c1da4b8341651"} Oct 30 02:07:06.577541 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:06.577516 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" event={"ID":"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3","Type":"ContainerStarted","Data":"85188ad811011e0c89f1a735ad5b702013f9c8cd8d09aa6d5ff59e30e3a0e464"} Oct 30 02:07:06.577604 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:06.577559 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" event={"ID":"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3","Type":"ContainerStarted","Data":"d5aef54cd74d9b5e352acbb97473e52e01e4bd20df8c42e980e4613959112867"} Oct 30 02:07:06.671108 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:06.670198 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" podStartSLOduration=2.670181791 podStartE2EDuration="2.670181791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:06.628572244 +0000 UTC m=+2256.207363325" watchObservedRunningTime="2026-10-30 02:07:06.670181791 +0000 UTC m=+2256.248972872" Oct 30 02:07:06.684787 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:06.684734 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" podStartSLOduration=2.6847153070000003 podStartE2EDuration="2.684715307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:06.678789469 +0000 UTC m=+2256.257580563" watchObservedRunningTime="2026-10-30 02:07:06.684715307 +0000 UTC m=+2256.263506389" Oct 30 02:07:07.096182 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:07.095041 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:07.314243 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:07.314207 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:07.323274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:07.322226 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 02:07:07.323605 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:07.323557 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" podUID="fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 02:07:07.334553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:07.334509 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:07.349390 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:07.349290 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 02:07:07.349675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:07.349631 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" podUID="f935c19c-9ed9-4f78-aa0a-d52823a3de7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 02:07:08.312112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:08.311604 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 02:07:08.312112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:08.311675 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" podUID="fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 02:07:08.333114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:08.327788 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 02:07:08.333114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:08.327861 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" podUID="f935c19c-9ed9-4f78-aa0a-d52823a3de7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 02:07:08.994153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:08.994075 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:09.317280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:09.317187 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 02:07:09.317852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:09.317819 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" podUID="fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 02:07:09.329235 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:09.329203 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 02:07:09.329496 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:09.329469 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" podUID="f935c19c-9ed9-4f78-aa0a-d52823a3de7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 02:07:10.271113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.270815 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4"] Oct 30 02:07:10.276437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.276404 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4" Oct 30 02:07:10.286441 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.286322 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4"] Oct 30 02:07:10.290047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.290022 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:07:10.298588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.298548 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88548125-0316-444c-bf04-fbc0fbe9bbd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8db6a5ee-cgwj4\" (UID: \"88548125-0316-444c-bf04-fbc0fbe9bbd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4" Oct 30 02:07:10.298728 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.298651 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88548125-0316-444c-bf04-fbc0fbe9bbd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8db6a5ee-cgwj4\" (UID: \"88548125-0316-444c-bf04-fbc0fbe9bbd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4" Oct 30 02:07:10.298859 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.298840 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq24v\" (UniqueName: \"kubernetes.io/projected/88548125-0316-444c-bf04-fbc0fbe9bbd4-kube-api-access-vq24v\") pod \"maas-api-revoke-sub-8db6a5ee-cgwj4\" (UID: \"88548125-0316-444c-bf04-fbc0fbe9bbd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4" Oct 30 02:07:10.313111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.304391 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:07:10.313111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.304487 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:07:10.313111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.304568 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:07:10.313111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.309494 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 02:07:10.313111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.309543 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" podUID="fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 02:07:10.355116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.353214 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 02:07:10.355116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.353277 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" podUID="f935c19c-9ed9-4f78-aa0a-d52823a3de7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 02:07:10.399713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.399676 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq24v\" (UniqueName: \"kubernetes.io/projected/88548125-0316-444c-bf04-fbc0fbe9bbd4-kube-api-access-vq24v\") pod \"maas-api-revoke-sub-8db6a5ee-cgwj4\" (UID: \"88548125-0316-444c-bf04-fbc0fbe9bbd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4" Oct 30 02:07:10.399910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.399783 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88548125-0316-444c-bf04-fbc0fbe9bbd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8db6a5ee-cgwj4\" (UID: \"88548125-0316-444c-bf04-fbc0fbe9bbd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4" Oct 30 02:07:10.423116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.408661 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88548125-0316-444c-bf04-fbc0fbe9bbd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8db6a5ee-cgwj4\" (UID: \"88548125-0316-444c-bf04-fbc0fbe9bbd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4" Oct 30 02:07:10.423116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.408737 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88548125-0316-444c-bf04-fbc0fbe9bbd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8db6a5ee-cgwj4\" (UID: \"88548125-0316-444c-bf04-fbc0fbe9bbd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4" Oct 30 02:07:10.463112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.461817 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq24v\" (UniqueName: \"kubernetes.io/projected/88548125-0316-444c-bf04-fbc0fbe9bbd4-kube-api-access-vq24v\") pod \"maas-api-revoke-sub-8db6a5ee-cgwj4\" (UID: \"88548125-0316-444c-bf04-fbc0fbe9bbd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4" Oct 30 02:07:10.463112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.462177 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88548125-0316-444c-bf04-fbc0fbe9bbd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8db6a5ee-cgwj4\" (UID: \"88548125-0316-444c-bf04-fbc0fbe9bbd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4" Oct 30 02:07:10.599073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.598991 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4" Oct 30 02:07:10.897189 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:10.896883 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:11.250109 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:11.250061 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4"] Oct 30 02:07:11.317122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:11.313553 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 02:07:11.317122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:11.313616 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" podUID="fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 02:07:11.337118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:11.334670 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 02:07:11.337118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:11.334730 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" podUID="f935c19c-9ed9-4f78-aa0a-d52823a3de7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 02:07:11.677154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:11.676927 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4" event={"ID":"88548125-0316-444c-bf04-fbc0fbe9bbd4","Type":"ContainerStarted","Data":"33e8b1d1a9d20eaaed9a0f87aabefe2f383fb4a31140d1d87a8f791723fd1fb1"} Oct 30 02:07:12.309193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:12.309153 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 02:07:12.309397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:12.309234 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" podUID="fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 02:07:12.327796 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:12.327753 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Oct 30 02:07:12.327958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:12.327804 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" podUID="f935c19c-9ed9-4f78-aa0a-d52823a3de7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Oct 30 02:07:12.366547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:12.366513 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:12.687311 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:12.687271 5998 generic.go:356] "Generic (PLEG): container finished" podID="88548125-0316-444c-bf04-fbc0fbe9bbd4" containerID="5e22f93ed625aac5270ad6fe4511877b4d3078238f8c4643c554a06de95bf35f" exitCode=0 Oct 30 02:07:12.687831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:12.687366 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4" event={"ID":"88548125-0316-444c-bf04-fbc0fbe9bbd4","Type":"ContainerDied","Data":"5e22f93ed625aac5270ad6fe4511877b4d3078238f8c4643c554a06de95bf35f"} Oct 30 02:07:13.449117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:13.446991 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:13.449117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:13.447705 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:13.453418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:13.453390 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:07:13.460775 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:13.460749 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:13.461131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:13.461084 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:13.462569 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:13.462546 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:07:13.873171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:13.873147 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4" Oct 30 02:07:13.988837 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:13.988806 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:13.996159 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:13.996133 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88548125-0316-444c-bf04-fbc0fbe9bbd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88548125-0316-444c-bf04-fbc0fbe9bbd4-maas-api-openshift-ca\") pod \"88548125-0316-444c-bf04-fbc0fbe9bbd4\" (UID: \"88548125-0316-444c-bf04-fbc0fbe9bbd4\") " Oct 30 02:07:13.996346 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:13.996228 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88548125-0316-444c-bf04-fbc0fbe9bbd4-kube-api-access-vq24v\" (UniqueName: \"kubernetes.io/projected/88548125-0316-444c-bf04-fbc0fbe9bbd4-kube-api-access-vq24v\") pod \"88548125-0316-444c-bf04-fbc0fbe9bbd4\" (UID: \"88548125-0316-444c-bf04-fbc0fbe9bbd4\") " Oct 30 02:07:13.996413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:13.996385 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88548125-0316-444c-bf04-fbc0fbe9bbd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88548125-0316-444c-bf04-fbc0fbe9bbd4-maas-api-xks-ca\") pod \"88548125-0316-444c-bf04-fbc0fbe9bbd4\" (UID: \"88548125-0316-444c-bf04-fbc0fbe9bbd4\") " Oct 30 02:07:13.996526 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:13.996503 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88548125-0316-444c-bf04-fbc0fbe9bbd4-maas-api-openshift-ca" pod "88548125-0316-444c-bf04-fbc0fbe9bbd4" (UID: "88548125-0316-444c-bf04-fbc0fbe9bbd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:07:13.996951 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:13.996908 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88548125-0316-444c-bf04-fbc0fbe9bbd4-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:07:13.998831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:13.998804 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88548125-0316-444c-bf04-fbc0fbe9bbd4-maas-api-xks-ca" pod "88548125-0316-444c-bf04-fbc0fbe9bbd4" (UID: "88548125-0316-444c-bf04-fbc0fbe9bbd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:07:13.998989 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:13.998856 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88548125-0316-444c-bf04-fbc0fbe9bbd4-kube-api-access-vq24v" pod "88548125-0316-444c-bf04-fbc0fbe9bbd4" (UID: "88548125-0316-444c-bf04-fbc0fbe9bbd4"). InnerVolumeSpecName "kube-api-access-vq24v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:14.098070 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:14.097977 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq24v\" (UniqueName: \"kubernetes.io/projected/88548125-0316-444c-bf04-fbc0fbe9bbd4-kube-api-access-vq24v\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:07:14.098369 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:14.098337 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88548125-0316-444c-bf04-fbc0fbe9bbd4-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:07:14.703581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:14.703548 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4" Oct 30 02:07:14.703581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:14.703561 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4" event={"ID":"88548125-0316-444c-bf04-fbc0fbe9bbd4","Type":"ContainerDied","Data":"33e8b1d1a9d20eaaed9a0f87aabefe2f383fb4a31140d1d87a8f791723fd1fb1"} Oct 30 02:07:14.703869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:14.703616 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33e8b1d1a9d20eaaed9a0f87aabefe2f383fb4a31140d1d87a8f791723fd1fb1" Oct 30 02:07:15.512813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:15.512763 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:15.619861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:15.619823 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:15.725574 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:15.725523 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:17.211807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:17.211767 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:18.695165 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:18.695129 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:07:18.930386 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:18.930343 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:19.926115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:19.922600 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:20.046114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:20.045957 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:21.085114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.084782 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:21.148113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.147252 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 02:07:21.219428 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.219384 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 02:07:21.259450 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.259385 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 02:07:21.261709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.261412 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 02:07:21.358601 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.356169 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 02:07:21.431209 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.429159 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:21.516115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.511140 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9"] Oct 30 02:07:21.516115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.511959 5998 cpu_manager.go:414] "containerMap: removing container" podUID="88548125-0316-444c-bf04-fbc0fbe9bbd4" containerName="revoke-keys" Oct 30 02:07:21.516115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.511981 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88548125-0316-444c-bf04-fbc0fbe9bbd4" containerName="revoke-keys" Oct 30 02:07:21.516115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.512210 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="88548125-0316-444c-bf04-fbc0fbe9bbd4" containerName="revoke-keys" Oct 30 02:07:21.615194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.606969 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:21.615606 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.615579 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9"] Oct 30 02:07:21.615895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.615878 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.656128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.633054 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-24fb97-openshift-default-dockercfg-stxdp\"" Oct 30 02:07:21.816809 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.816761 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-token\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.817109 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.817057 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-data\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.817413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.817320 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-credential-socket\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.817728 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.817709 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-podinfo\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.817989 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.817964 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/820068d2-ba3f-4127-8f01-3b21c61acebd-istiod-ca-cert\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.818154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.818128 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnq8g\" (UniqueName: \"kubernetes.io/projected/820068d2-ba3f-4127-8f01-3b21c61acebd-kube-api-access-nnq8g\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.818374 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.818355 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-workload-socket\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.818583 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.818563 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-envoy\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.818744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.818727 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-workload-certs\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.917208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.916362 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:21.920957 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.920874 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/820068d2-ba3f-4127-8f01-3b21c61acebd-istiod-ca-cert\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.922028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.922004 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnq8g\" (UniqueName: \"kubernetes.io/projected/820068d2-ba3f-4127-8f01-3b21c61acebd-kube-api-access-nnq8g\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.922620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.922596 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-workload-socket\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.923254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.923235 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-envoy\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.923776 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.923756 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-workload-certs\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.924298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.924278 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-token\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.924744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.924724 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-data\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.925206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.925185 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-credential-socket\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.925774 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.925753 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-podinfo\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.949582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.923139 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-workload-socket\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.965126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.925127 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-data\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.965399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.924170 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-workload-certs\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.965572 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.925557 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-credential-socket\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.965716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.921937 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/820068d2-ba3f-4127-8f01-3b21c61acebd-istiod-ca-cert\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.982992 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.982921 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-token\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.983833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.983807 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-podinfo\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:21.984281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:21.984233 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-envoy\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:22.002740 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:22.002662 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnq8g\" (UniqueName: \"kubernetes.io/projected/820068d2-ba3f-4127-8f01-3b21c61acebd-kube-api-access-nnq8g\") pod \"e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:22.068120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:22.067451 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:22.444779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:22.444754 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9"] Oct 30 02:07:22.449810 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:07:22.449777 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820068d2_ba3f_4127_8f01_3b21c61acebd.slice/crio-9be56fcfb5d4dd4d2fd9daea6b91c06d140a5d6917874f9d435125b1ed9c1ffb WatchSource:0}: Error finding container 9be56fcfb5d4dd4d2fd9daea6b91c06d140a5d6917874f9d435125b1ed9c1ffb: Status 404 returned error can't find the container with id 9be56fcfb5d4dd4d2fd9daea6b91c06d140a5d6917874f9d435125b1ed9c1ffb Oct 30 02:07:22.452723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:22.452688 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:07:22.452798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:22.452752 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:07:22.452798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:22.452781 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:07:22.920113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:22.917844 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" event={"ID":"820068d2-ba3f-4127-8f01-3b21c61acebd","Type":"ContainerStarted","Data":"6d618c4cef412e22402f92cbb9bba4fc0122b35c48d9dd47351e462a728d8c76"} Oct 30 02:07:22.920113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:22.917957 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" event={"ID":"820068d2-ba3f-4127-8f01-3b21c61acebd","Type":"ContainerStarted","Data":"9be56fcfb5d4dd4d2fd9daea6b91c06d140a5d6917874f9d435125b1ed9c1ffb"} Oct 30 02:07:23.044111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:23.030182 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" podStartSLOduration=2.030155969 podStartE2EDuration="2.030155969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:22.992604766 +0000 UTC m=+2272.571395848" watchObservedRunningTime="2026-10-30 02:07:23.030155969 +0000 UTC m=+2272.608947050" Oct 30 02:07:23.077114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:23.076011 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:23.077319 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:23.077241 5998 patch_prober.go:29] interesting pod/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 02:07:23.077371 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:23.077314 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" podUID="820068d2-ba3f-4127-8f01-3b21c61acebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 02:07:23.173200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:23.172577 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:23.282326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:23.282288 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:24.073847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:24.073819 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:24.937410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:24.937378 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:24.938552 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:24.938528 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:07:25.083221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:25.083187 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:25.227690 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:07:25.227599 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:07:25.881329 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:25.881297 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:26.013509 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:26.013472 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:26.500558 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:26.500521 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:26.922659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:26.922616 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:27.562806 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:27.562772 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:27.674717 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:27.674681 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:28.262018 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:28.261986 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:28.697429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:28.697394 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:29.195843 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:29.195807 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:29.895387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:29.895354 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv"] Oct 30 02:07:29.899477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:29.899458 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv" Oct 30 02:07:29.919814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:29.919787 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv"] Oct 30 02:07:29.927027 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:29.926993 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:07:29.927311 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:29.927281 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:07:30.006932 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:30.006903 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:07:30.007175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:30.007123 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:07:30.036889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:30.036850 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a541f1e-8c33-4f14-95f1-98ec0f568665-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20e36cb2-6wmpv\" (UID: \"1a541f1e-8c33-4f14-95f1-98ec0f568665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv" Oct 30 02:07:30.037044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:30.036999 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f74lm\" (UniqueName: \"kubernetes.io/projected/1a541f1e-8c33-4f14-95f1-98ec0f568665-kube-api-access-f74lm\") pod \"maas-api-revoke-sub-20e36cb2-6wmpv\" (UID: \"1a541f1e-8c33-4f14-95f1-98ec0f568665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv" Oct 30 02:07:30.037147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:30.037111 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a541f1e-8c33-4f14-95f1-98ec0f568665-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20e36cb2-6wmpv\" (UID: \"1a541f1e-8c33-4f14-95f1-98ec0f568665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv" Oct 30 02:07:30.139013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:30.138969 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f74lm\" (UniqueName: \"kubernetes.io/projected/1a541f1e-8c33-4f14-95f1-98ec0f568665-kube-api-access-f74lm\") pod \"maas-api-revoke-sub-20e36cb2-6wmpv\" (UID: \"1a541f1e-8c33-4f14-95f1-98ec0f568665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv" Oct 30 02:07:30.139269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:30.139059 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a541f1e-8c33-4f14-95f1-98ec0f568665-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20e36cb2-6wmpv\" (UID: \"1a541f1e-8c33-4f14-95f1-98ec0f568665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv" Oct 30 02:07:30.139323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:30.139282 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a541f1e-8c33-4f14-95f1-98ec0f568665-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20e36cb2-6wmpv\" (UID: \"1a541f1e-8c33-4f14-95f1-98ec0f568665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv" Oct 30 02:07:30.139939 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:30.139915 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a541f1e-8c33-4f14-95f1-98ec0f568665-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20e36cb2-6wmpv\" (UID: \"1a541f1e-8c33-4f14-95f1-98ec0f568665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv" Oct 30 02:07:30.141675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:30.141649 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a541f1e-8c33-4f14-95f1-98ec0f568665-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20e36cb2-6wmpv\" (UID: \"1a541f1e-8c33-4f14-95f1-98ec0f568665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv" Oct 30 02:07:30.163499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:30.163467 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f74lm\" (UniqueName: \"kubernetes.io/projected/1a541f1e-8c33-4f14-95f1-98ec0f568665-kube-api-access-f74lm\") pod \"maas-api-revoke-sub-20e36cb2-6wmpv\" (UID: \"1a541f1e-8c33-4f14-95f1-98ec0f568665\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv" Oct 30 02:07:30.213801 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:30.213759 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv" Oct 30 02:07:30.521814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:30.521659 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:30.569863 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:07:30.569824 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a541f1e_8c33_4f14_95f1_98ec0f568665.slice/crio-a6f833376b60cc6acceb715b01974e923a7cd17b17d8d48302e6ca66c8f84c7a WatchSource:0}: Error finding container a6f833376b60cc6acceb715b01974e923a7cd17b17d8d48302e6ca66c8f84c7a: Status 404 returned error can't find the container with id a6f833376b60cc6acceb715b01974e923a7cd17b17d8d48302e6ca66c8f84c7a Oct 30 02:07:30.575105 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:30.575064 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv"] Oct 30 02:07:30.995601 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:30.995550 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv" event={"ID":"1a541f1e-8c33-4f14-95f1-98ec0f568665","Type":"ContainerStarted","Data":"84a4739e0da7e435d277c9a0d7732550d1459a8c3f73203cee85dec1cb873174"} Oct 30 02:07:30.996261 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:30.995632 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv" event={"ID":"1a541f1e-8c33-4f14-95f1-98ec0f568665","Type":"ContainerStarted","Data":"a6f833376b60cc6acceb715b01974e923a7cd17b17d8d48302e6ca66c8f84c7a"} Oct 30 02:07:31.037902 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:31.037844 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv" podStartSLOduration=2.037827428 podStartE2EDuration="2.037827428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:31.032645734 +0000 UTC m=+2280.611436815" watchObservedRunningTime="2026-10-30 02:07:31.037827428 +0000 UTC m=+2280.616618508" Oct 30 02:07:31.299791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:31.299754 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:31.406013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:31.405933 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:31.965009 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:31.964975 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:32.004675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:32.004635 5998 generic.go:356] "Generic (PLEG): container finished" podID="1a541f1e-8c33-4f14-95f1-98ec0f568665" containerID="84a4739e0da7e435d277c9a0d7732550d1459a8c3f73203cee85dec1cb873174" exitCode=0 Oct 30 02:07:32.005275 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:32.004729 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv" event={"ID":"1a541f1e-8c33-4f14-95f1-98ec0f568665","Type":"ContainerDied","Data":"84a4739e0da7e435d277c9a0d7732550d1459a8c3f73203cee85dec1cb873174"} Oct 30 02:07:32.439283 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:32.439228 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:33.171722 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:33.171688 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv" Oct 30 02:07:33.311151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:33.311022 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a541f1e-8c33-4f14-95f1-98ec0f568665-kube-api-access-f74lm\" (UniqueName: \"kubernetes.io/projected/1a541f1e-8c33-4f14-95f1-98ec0f568665-kube-api-access-f74lm\") pod \"1a541f1e-8c33-4f14-95f1-98ec0f568665\" (UID: \"1a541f1e-8c33-4f14-95f1-98ec0f568665\") " Oct 30 02:07:33.311151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:33.311071 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a541f1e-8c33-4f14-95f1-98ec0f568665-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a541f1e-8c33-4f14-95f1-98ec0f568665-maas-api-xks-ca\") pod \"1a541f1e-8c33-4f14-95f1-98ec0f568665\" (UID: \"1a541f1e-8c33-4f14-95f1-98ec0f568665\") " Oct 30 02:07:33.311393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:33.311317 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a541f1e-8c33-4f14-95f1-98ec0f568665-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a541f1e-8c33-4f14-95f1-98ec0f568665-maas-api-openshift-ca\") pod \"1a541f1e-8c33-4f14-95f1-98ec0f568665\" (UID: \"1a541f1e-8c33-4f14-95f1-98ec0f568665\") " Oct 30 02:07:33.311729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:33.311691 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a541f1e-8c33-4f14-95f1-98ec0f568665-maas-api-openshift-ca" pod "1a541f1e-8c33-4f14-95f1-98ec0f568665" (UID: "1a541f1e-8c33-4f14-95f1-98ec0f568665"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:07:33.311904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:33.311886 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a541f1e-8c33-4f14-95f1-98ec0f568665-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:07:33.313367 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:33.313339 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a541f1e-8c33-4f14-95f1-98ec0f568665-maas-api-xks-ca" pod "1a541f1e-8c33-4f14-95f1-98ec0f568665" (UID: "1a541f1e-8c33-4f14-95f1-98ec0f568665"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:07:33.313703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:33.313678 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a541f1e-8c33-4f14-95f1-98ec0f568665-kube-api-access-f74lm" pod "1a541f1e-8c33-4f14-95f1-98ec0f568665" (UID: "1a541f1e-8c33-4f14-95f1-98ec0f568665"). InnerVolumeSpecName "kube-api-access-f74lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:33.412939 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:33.412902 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f74lm\" (UniqueName: \"kubernetes.io/projected/1a541f1e-8c33-4f14-95f1-98ec0f568665-kube-api-access-f74lm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:07:33.412939 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:33.412935 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a541f1e-8c33-4f14-95f1-98ec0f568665-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:07:34.023103 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:34.023058 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv" Oct 30 02:07:34.023285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:34.023125 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv" event={"ID":"1a541f1e-8c33-4f14-95f1-98ec0f568665","Type":"ContainerDied","Data":"a6f833376b60cc6acceb715b01974e923a7cd17b17d8d48302e6ca66c8f84c7a"} Oct 30 02:07:34.023285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:34.023186 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6f833376b60cc6acceb715b01974e923a7cd17b17d8d48302e6ca66c8f84c7a" Oct 30 02:07:34.161917 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:34.161877 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:35.028716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:35.028672 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:35.246958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:35.246926 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:35.360056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:35.359955 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:36.172453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:36.172070 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:36.418209 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:36.418170 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:37.875758 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:37.875728 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:38.928441 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:38.928401 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:39.608661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:39.608624 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:41.116801 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:41.116761 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:41.663794 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:41.663729 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:41.789866 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:41.789832 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:42.200956 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:42.200914 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:42.371112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:42.366010 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:42.712121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:42.709071 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677"] Oct 30 02:07:42.712121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:42.709363 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" podUID="592f8d79-2a33-4c5b-9639-c60bd7aedf9f" containerName="istio-proxy" containerID="cri-o://de7805abc6adf67ef4d005dd78bc212f8e532d9e870b77cbb3cb0acb31c5bf38" gracePeriod=30 Oct 30 02:07:43.167114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:43.165626 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:43.913930 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:43.913872 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:45.579149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:45.579085 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:45.847333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:45.847240 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:46.691142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:46.691110 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:46.806590 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:46.806554 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:47.632982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:47.632948 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:47.635940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:47.635908 5998 patch_prober.go:29] interesting pod/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:07:47.636120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:47.635976 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" podUID="592f8d79-2a33-4c5b-9639-c60bd7aedf9f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:07:47.760763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:47.760727 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:47.958694 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:47.958663 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 02:07:48.075477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.075442 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-workload-certs\") pod \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " Oct 30 02:07:48.075675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.075503 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istiod-ca-cert\") pod \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " Oct 30 02:07:48.075675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.075554 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-token\" (UniqueName: \"kubernetes.io/projected/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-token\") pod \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " Oct 30 02:07:48.075675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.075580 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-data\") pod \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " Oct 30 02:07:48.075675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.075624 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-podinfo\") pod \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " Oct 30 02:07:48.075893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.075692 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-envoy\") pod \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " Oct 30 02:07:48.075893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.075770 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-workload-certs" pod "592f8d79-2a33-4c5b-9639-c60bd7aedf9f" (UID: "592f8d79-2a33-4c5b-9639-c60bd7aedf9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:48.075893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.075787 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-credential-socket\") pod \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " Oct 30 02:07:48.075893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.075824 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-workload-socket\") pod \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " Oct 30 02:07:48.076179 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.075979 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istiod-ca-cert" pod "592f8d79-2a33-4c5b-9639-c60bd7aedf9f" (UID: "592f8d79-2a33-4c5b-9639-c60bd7aedf9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:07:48.076179 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.076038 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-kube-api-access-mpp4x\" (UniqueName: \"kubernetes.io/projected/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-kube-api-access-mpp4x\") pod \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\" (UID: \"592f8d79-2a33-4c5b-9639-c60bd7aedf9f\") " Oct 30 02:07:48.076321 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.076296 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-credential-socket" pod "592f8d79-2a33-4c5b-9639-c60bd7aedf9f" (UID: "592f8d79-2a33-4c5b-9639-c60bd7aedf9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:48.076590 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.076544 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-workload-socket" pod "592f8d79-2a33-4c5b-9639-c60bd7aedf9f" (UID: "592f8d79-2a33-4c5b-9639-c60bd7aedf9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:48.076590 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.076563 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-data" pod "592f8d79-2a33-4c5b-9639-c60bd7aedf9f" (UID: "592f8d79-2a33-4c5b-9639-c60bd7aedf9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:48.076839 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.076818 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:07:48.076878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.076845 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:07:48.076878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.076858 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:07:48.076878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.076871 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:07:48.076987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.076886 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:07:48.078727 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.078705 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-podinfo" pod "592f8d79-2a33-4c5b-9639-c60bd7aedf9f" (UID: "592f8d79-2a33-4c5b-9639-c60bd7aedf9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:07:48.078727 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.078712 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-token" pod "592f8d79-2a33-4c5b-9639-c60bd7aedf9f" (UID: "592f8d79-2a33-4c5b-9639-c60bd7aedf9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:48.078864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.078725 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-envoy" pod "592f8d79-2a33-4c5b-9639-c60bd7aedf9f" (UID: "592f8d79-2a33-4c5b-9639-c60bd7aedf9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:07:48.078864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.078741 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-kube-api-access-mpp4x" pod "592f8d79-2a33-4c5b-9639-c60bd7aedf9f" (UID: "592f8d79-2a33-4c5b-9639-c60bd7aedf9f"). InnerVolumeSpecName "kube-api-access-mpp4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:48.178290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.178260 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpp4x\" (UniqueName: \"kubernetes.io/projected/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-kube-api-access-mpp4x\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:07:48.178290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.178287 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:07:48.178290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.178297 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:07:48.178547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.178305 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/592f8d79-2a33-4c5b-9639-c60bd7aedf9f-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:07:48.206470 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.206375 5998 generic.go:356] "Generic (PLEG): container finished" podID="592f8d79-2a33-4c5b-9639-c60bd7aedf9f" containerID="de7805abc6adf67ef4d005dd78bc212f8e532d9e870b77cbb3cb0acb31c5bf38" exitCode=0 Oct 30 02:07:48.206470 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.206447 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" Oct 30 02:07:48.206470 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.206457 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" event={"ID":"592f8d79-2a33-4c5b-9639-c60bd7aedf9f","Type":"ContainerDied","Data":"de7805abc6adf67ef4d005dd78bc212f8e532d9e870b77cbb3cb0acb31c5bf38"} Oct 30 02:07:48.206744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.206525 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677" event={"ID":"592f8d79-2a33-4c5b-9639-c60bd7aedf9f","Type":"ContainerDied","Data":"ae27310ee77b367dcb73d0ff45d2d21a237ece416106b1365faa3a76df4fdd84"} Oct 30 02:07:48.206744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.206549 5998 scope.go:122] "RemoveContainer" containerID="de7805abc6adf67ef4d005dd78bc212f8e532d9e870b77cbb3cb0acb31c5bf38" Oct 30 02:07:48.220709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.220655 5998 scope.go:122] "RemoveContainer" containerID="de7805abc6adf67ef4d005dd78bc212f8e532d9e870b77cbb3cb0acb31c5bf38" Oct 30 02:07:48.221030 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:07:48.221010 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de7805abc6adf67ef4d005dd78bc212f8e532d9e870b77cbb3cb0acb31c5bf38\": container with ID starting with de7805abc6adf67ef4d005dd78bc212f8e532d9e870b77cbb3cb0acb31c5bf38 not found: ID does not exist" containerID="de7805abc6adf67ef4d005dd78bc212f8e532d9e870b77cbb3cb0acb31c5bf38" Oct 30 02:07:48.221167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.221043 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de7805abc6adf67ef4d005dd78bc212f8e532d9e870b77cbb3cb0acb31c5bf38"} err="failed to get container status \"de7805abc6adf67ef4d005dd78bc212f8e532d9e870b77cbb3cb0acb31c5bf38\": rpc error: code = NotFound desc = could not find container \"de7805abc6adf67ef4d005dd78bc212f8e532d9e870b77cbb3cb0acb31c5bf38\": container with ID starting with de7805abc6adf67ef4d005dd78bc212f8e532d9e870b77cbb3cb0acb31c5bf38 not found: ID does not exist" Oct 30 02:07:48.258540 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.258510 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677"] Oct 30 02:07:48.262349 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.262320 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-5p677"] Oct 30 02:07:48.541577 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:48.541472 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="592f8d79-2a33-4c5b-9639-c60bd7aedf9f" path="/var/lib/kubelet/pods/592f8d79-2a33-4c5b-9639-c60bd7aedf9f/volumes" Oct 30 02:07:49.539264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:49.539222 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:49.792632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:49.792535 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:49.979936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:49.979902 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:50.404981 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.404933 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:50.445715 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.445679 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl"] Oct 30 02:07:50.446359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.446333 5998 cpu_manager.go:414] "containerMap: removing container" podUID="592f8d79-2a33-4c5b-9639-c60bd7aedf9f" containerName="istio-proxy" Oct 30 02:07:50.446359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.446355 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="592f8d79-2a33-4c5b-9639-c60bd7aedf9f" containerName="istio-proxy" Oct 30 02:07:50.446573 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.446409 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1a541f1e-8c33-4f14-95f1-98ec0f568665" containerName="revoke-keys" Oct 30 02:07:50.446573 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.446418 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a541f1e-8c33-4f14-95f1-98ec0f568665" containerName="revoke-keys" Oct 30 02:07:50.446680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.446574 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a541f1e-8c33-4f14-95f1-98ec0f568665" containerName="revoke-keys" Oct 30 02:07:50.446680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.446590 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="592f8d79-2a33-4c5b-9639-c60bd7aedf9f" containerName="istio-proxy" Oct 30 02:07:50.451908 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.451879 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.466653 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.466627 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-9a1cd0-openshift-default-dockercfg-mww6v\"" Oct 30 02:07:50.482718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.482684 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl"] Oct 30 02:07:50.517628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.517592 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-workload-certs\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.517827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.517636 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b95gv\" (UniqueName: \"kubernetes.io/projected/5220aca0-19d7-444c-b92b-da5361f306dc-kube-api-access-b95gv\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.517827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.517730 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5220aca0-19d7-444c-b92b-da5361f306dc-istio-token\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.517827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.517767 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5220aca0-19d7-444c-b92b-da5361f306dc-istio-podinfo\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.517827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.517799 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-workload-socket\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.518065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.517923 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-credential-socket\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.518065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.517943 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-istio-data\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.518065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.518051 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-istio-envoy\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.518287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.518075 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5220aca0-19d7-444c-b92b-da5361f306dc-istiod-ca-cert\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.524842 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.524805 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:50.619262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.619215 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-credential-socket\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.619725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.619269 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-istio-data\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.619725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.619377 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-istio-envoy\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.619725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.619406 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5220aca0-19d7-444c-b92b-da5361f306dc-istiod-ca-cert\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.619725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.619632 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-credential-socket\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.619725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.619686 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-istio-data\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.620134 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.620106 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-workload-certs\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.620367 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.620337 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b95gv\" (UniqueName: \"kubernetes.io/projected/5220aca0-19d7-444c-b92b-da5361f306dc-kube-api-access-b95gv\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.620468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.620411 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5220aca0-19d7-444c-b92b-da5361f306dc-istio-token\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.620468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.620343 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-workload-certs\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.620468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.620441 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5220aca0-19d7-444c-b92b-da5361f306dc-istio-podinfo\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.620642 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.620487 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-workload-socket\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.620642 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.620225 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5220aca0-19d7-444c-b92b-da5361f306dc-istiod-ca-cert\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.620891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.620871 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-workload-socket\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.621913 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.621886 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-istio-envoy\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.622931 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.622911 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5220aca0-19d7-444c-b92b-da5361f306dc-istio-podinfo\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.642298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.642262 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5220aca0-19d7-444c-b92b-da5361f306dc-istio-token\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.643130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.643071 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b95gv\" (UniqueName: \"kubernetes.io/projected/5220aca0-19d7-444c-b92b-da5361f306dc-kube-api-access-b95gv\") pod \"e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:50.645858 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.645825 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:50.781031 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:50.780366 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:51.260239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:51.260215 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl"] Oct 30 02:07:51.263595 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:07:51.263564 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5220aca0_19d7_444c_b92b_da5361f306dc.slice/crio-a51dd581415eaa3ec933979098c146949c07db59e13034b4a6ef871f346c0c4b WatchSource:0}: Error finding container a51dd581415eaa3ec933979098c146949c07db59e13034b4a6ef871f346c0c4b: Status 404 returned error can't find the container with id a51dd581415eaa3ec933979098c146949c07db59e13034b4a6ef871f346c0c4b Oct 30 02:07:51.266626 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:51.266593 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:07:51.266755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:51.266671 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:07:51.266755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:51.266719 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:07:51.799289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:51.799237 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:52.242323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:52.242285 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" event={"ID":"5220aca0-19d7-444c-b92b-da5361f306dc","Type":"ContainerStarted","Data":"178b62369ec547602268c4a39da502ed1f1a1d42b6ffddfc1a60baacbbbcf2a2"} Oct 30 02:07:52.242550 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:52.242348 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" event={"ID":"5220aca0-19d7-444c-b92b-da5361f306dc","Type":"ContainerStarted","Data":"a51dd581415eaa3ec933979098c146949c07db59e13034b4a6ef871f346c0c4b"} Oct 30 02:07:52.309556 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:52.309492 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" podStartSLOduration=2.309473293 podStartE2EDuration="2.309473293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:52.297783495 +0000 UTC m=+2301.876574576" watchObservedRunningTime="2026-10-30 02:07:52.309473293 +0000 UTC m=+2301.888264375" Oct 30 02:07:52.781712 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:52.781675 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:52.787276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:52.787250 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:53.043385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:53.043301 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:53.216568 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:53.216530 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:53.255201 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:53.255160 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:53.256197 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:53.256168 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:07:53.335160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:53.334976 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:53.596868 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:53.596734 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:54.792974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:54.792934 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:55.089554 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:55.089465 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:55.211906 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:55.211869 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:55.222272 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:07:55.222223 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:07:56.740112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:56.740070 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:56.849466 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:56.849422 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:07:58.466791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:07:58.466758 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:00.008325 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:00.008288 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:00.942121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:00.941651 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:08:00.987785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:00.987722 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:01.102648 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:01.102615 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:01.787331 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:01.787291 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:02.023567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:02.023534 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:02.290931 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:02.290832 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:03.055861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:03.055823 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:03.257962 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:03.257914 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:03.373169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:03.373073 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:03.971061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:03.971001 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:04.144666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:04.144635 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:04.350352 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:04.350258 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:04.359661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:04.357855 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td"] Oct 30 02:08:04.359661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:04.358135 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" podUID="0ad87fa4-78c9-4e60-8706-a78dbd1002b1" containerName="istio-proxy" containerID="cri-o://0bcabb375dd5f7274f7f3a9fe236f7f6e60d108a3e1e7c73b1e7e676110e270d" gracePeriod=30 Oct 30 02:08:05.130980 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:05.130948 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:05.928403 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:05.928371 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:06.586245 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:06.586209 5998 patch_prober.go:29] interesting pod/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:08:06.586811 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:06.586273 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" podUID="0ad87fa4-78c9-4e60-8706-a78dbd1002b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:08:07.643833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:07.643758 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:07.778303 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:07.778267 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:07.892933 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:07.892898 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:09.482276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.482237 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:09.670540 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.670239 5998 generic.go:356] "Generic (PLEG): container finished" podID="0ad87fa4-78c9-4e60-8706-a78dbd1002b1" containerID="0bcabb375dd5f7274f7f3a9fe236f7f6e60d108a3e1e7c73b1e7e676110e270d" exitCode=0 Oct 30 02:08:09.670540 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.670354 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" event={"ID":"0ad87fa4-78c9-4e60-8706-a78dbd1002b1","Type":"ContainerDied","Data":"0bcabb375dd5f7274f7f3a9fe236f7f6e60d108a3e1e7c73b1e7e676110e270d"} Oct 30 02:08:09.670540 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.670411 5998 scope.go:122] "RemoveContainer" containerID="e806753998881e2282853e62697af3fffc9de7171f25c01c8f0f3d17b6c1fc08" Oct 30 02:08:09.697185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.697148 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:08:09.848385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.846802 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istiod-ca-cert\") pod \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " Oct 30 02:08:09.848385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.846912 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-credential-socket\") pod \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " Oct 30 02:08:09.848385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.846983 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-workload-socket\") pod \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " Oct 30 02:08:09.848385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.847056 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-envoy\") pod \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " Oct 30 02:08:09.848385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.847117 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-podinfo\") pod \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " Oct 30 02:08:09.848385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.847148 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-token\" (UniqueName: \"kubernetes.io/projected/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-token\") pod \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " Oct 30 02:08:09.848385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.847268 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-data\") pod \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " Oct 30 02:08:09.848385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.847330 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-workload-certs\") pod \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " Oct 30 02:08:09.848385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.847369 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-kube-api-access-6grrz\" (UniqueName: \"kubernetes.io/projected/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-kube-api-access-6grrz\") pod \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\" (UID: \"0ad87fa4-78c9-4e60-8706-a78dbd1002b1\") " Oct 30 02:08:09.849874 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.849330 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-credential-socket" pod "0ad87fa4-78c9-4e60-8706-a78dbd1002b1" (UID: "0ad87fa4-78c9-4e60-8706-a78dbd1002b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:09.849874 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.849547 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-workload-socket" pod "0ad87fa4-78c9-4e60-8706-a78dbd1002b1" (UID: "0ad87fa4-78c9-4e60-8706-a78dbd1002b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:09.849874 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.849590 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istiod-ca-cert" pod "0ad87fa4-78c9-4e60-8706-a78dbd1002b1" (UID: "0ad87fa4-78c9-4e60-8706-a78dbd1002b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:08:09.850158 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.849907 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-workload-certs" pod "0ad87fa4-78c9-4e60-8706-a78dbd1002b1" (UID: "0ad87fa4-78c9-4e60-8706-a78dbd1002b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:09.850158 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.849859 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-data" pod "0ad87fa4-78c9-4e60-8706-a78dbd1002b1" (UID: "0ad87fa4-78c9-4e60-8706-a78dbd1002b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:09.850755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.850727 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-envoy" pod "0ad87fa4-78c9-4e60-8706-a78dbd1002b1" (UID: "0ad87fa4-78c9-4e60-8706-a78dbd1002b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:09.851988 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.851964 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-kube-api-access-6grrz" pod "0ad87fa4-78c9-4e60-8706-a78dbd1002b1" (UID: "0ad87fa4-78c9-4e60-8706-a78dbd1002b1"). InnerVolumeSpecName "kube-api-access-6grrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:09.852411 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.852387 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-podinfo" pod "0ad87fa4-78c9-4e60-8706-a78dbd1002b1" (UID: "0ad87fa4-78c9-4e60-8706-a78dbd1002b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:08:09.852626 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.852606 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-token" pod "0ad87fa4-78c9-4e60-8706-a78dbd1002b1" (UID: "0ad87fa4-78c9-4e60-8706-a78dbd1002b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:09.949723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.949682 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:09.949723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.949723 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:09.949965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.949738 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:09.949965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.949751 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:09.949965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.949764 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:09.949965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.949776 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:09.949965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.949787 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:09.949965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.949799 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:09.949965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:09.949812 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6grrz\" (UniqueName: \"kubernetes.io/projected/0ad87fa4-78c9-4e60-8706-a78dbd1002b1-kube-api-access-6grrz\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:10.580681 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:10.580629 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:10.688628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:10.688587 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" event={"ID":"0ad87fa4-78c9-4e60-8706-a78dbd1002b1","Type":"ContainerDied","Data":"0d316c49ed3057d9a207ac016c55f2d26d6a7af2dbf7b00abf071bb5e77c0351"} Oct 30 02:08:10.689040 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:10.688998 5998 scope.go:122] "RemoveContainer" containerID="0bcabb375dd5f7274f7f3a9fe236f7f6e60d108a3e1e7c73b1e7e676110e270d" Oct 30 02:08:10.689376 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:10.689355 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td" Oct 30 02:08:10.695364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:10.695327 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:10.782926 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:10.782891 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td"] Oct 30 02:08:10.788147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:10.788097 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-2979be-openshift-default-5d449c749-4t7td"] Oct 30 02:08:10.808925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:10.808890 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:10.979325 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:10.979275 5998 patch_prober.go:29] interesting pod/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:08:10.979497 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:10.979347 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" podUID="820068d2-ba3f-4127-8f01-3b21c61acebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:08:11.423374 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:11.423316 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:11.543556 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:11.543520 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:12.542965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:12.542883 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ad87fa4-78c9-4e60-8706-a78dbd1002b1" path="/var/lib/kubelet/pods/0ad87fa4-78c9-4e60-8706-a78dbd1002b1/volumes" Oct 30 02:08:12.643500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:12.643399 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:12.708961 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:12.708928 5998 generic.go:356] "Generic (PLEG): container finished" podID="820068d2-ba3f-4127-8f01-3b21c61acebd" containerID="6d618c4cef412e22402f92cbb9bba4fc0122b35c48d9dd47351e462a728d8c76" exitCode=0 Oct 30 02:08:12.709159 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:12.709006 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" event={"ID":"820068d2-ba3f-4127-8f01-3b21c61acebd","Type":"ContainerDied","Data":"6d618c4cef412e22402f92cbb9bba4fc0122b35c48d9dd47351e462a728d8c76"} Oct 30 02:08:12.709497 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:12.709482 5998 scope.go:122] "RemoveContainer" containerID="6d618c4cef412e22402f92cbb9bba4fc0122b35c48d9dd47351e462a728d8c76" Oct 30 02:08:12.711451 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:12.711415 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:08:12.711554 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:12.711509 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:08:12.711608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:12.711561 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:08:13.245990 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:13.245953 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:13.719594 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:13.719555 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" event={"ID":"820068d2-ba3f-4127-8f01-3b21c61acebd","Type":"ContainerStarted","Data":"122a307ed172d34a716059cf3da2d2e8f928796402ad64b1a67c55b0e3dc6084"} Oct 30 02:08:14.067923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:14.067831 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:08:14.068215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:14.068193 5998 patch_prober.go:29] interesting pod/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 02:08:14.068271 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:14.068241 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" podUID="820068d2-ba3f-4127-8f01-3b21c61acebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 02:08:14.962717 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:14.962678 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:15.069800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:15.069757 5998 patch_prober.go:29] interesting pod/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 02:08:15.070016 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:15.069837 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" podUID="820068d2-ba3f-4127-8f01-3b21c61acebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 02:08:16.069161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:16.069121 5998 patch_prober.go:29] interesting pod/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 02:08:16.069583 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:16.069197 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" podUID="820068d2-ba3f-4127-8f01-3b21c61acebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 02:08:16.557935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:16.557899 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:16.673814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:16.673774 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:16.791630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:16.791594 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:17.068163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:17.068085 5998 patch_prober.go:29] interesting pod/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 02:08:17.068386 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:17.068195 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" podUID="820068d2-ba3f-4127-8f01-3b21c61acebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 02:08:17.073337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:17.073306 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:17.680034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:17.679999 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:17.814272 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:17.814234 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:18.068550 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:18.068456 5998 patch_prober.go:29] interesting pod/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 02:08:18.068849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:18.068821 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" podUID="820068d2-ba3f-4127-8f01-3b21c61acebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 02:08:18.450767 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:18.450729 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:19.068749 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:19.068711 5998 patch_prober.go:29] interesting pod/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 02:08:19.068926 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:19.068780 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" podUID="820068d2-ba3f-4127-8f01-3b21c61acebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 02:08:19.580449 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:19.580411 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:20.073370 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:20.073331 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:08:20.074152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:20.074123 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:08:20.075078 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:20.075055 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:08:20.121312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:20.121283 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:21.301718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:21.301663 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:21.417058 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:21.417022 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:21.536393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:21.536275 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:22.398813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:22.398778 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:22.999038 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:22.999003 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:23.126342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:23.126307 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:23.972479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:23.972441 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt"] Oct 30 02:08:23.972959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:23.972656 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" podUID="6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" containerName="istio-proxy" containerID="cri-o://3a734fa25185c308dc92d15f47078ab76d4e137255a5dea9af10a5b83172e6b2" gracePeriod=30 Oct 30 02:08:24.402475 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:24.402373 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:25.226299 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:08:25.226258 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:08:25.981272 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:25.981238 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:26.555845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:26.555812 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:26.700750 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:26.700712 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:27.218398 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:27.218324 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:27.447165 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:27.447110 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:27.572211 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:27.572118 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s"] Oct 30 02:08:27.573024 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:27.572993 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" podUID="b24b4bd4-4eb0-412c-a54c-250a72795d94" containerName="istio-proxy" containerID="cri-o://065632ed5247eef5acd54bcd4df4e5ad60b7fcccb360d758a5adf03310af8ff8" gracePeriod=30 Oct 30 02:08:27.601608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:27.601579 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:28.277652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:28.277613 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:29.314558 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.314083 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 02:08:29.461453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.461418 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-podinfo\") pod \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " Oct 30 02:08:29.461630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.461466 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-kube-api-access-68wgw\" (UniqueName: \"kubernetes.io/projected/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-kube-api-access-68wgw\") pod \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " Oct 30 02:08:29.461630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.461532 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-data\") pod \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " Oct 30 02:08:29.461630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.461576 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-token\" (UniqueName: \"kubernetes.io/projected/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-token\") pod \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " Oct 30 02:08:29.461630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.461616 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-workload-certs\") pod \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " Oct 30 02:08:29.461857 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.461698 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istiod-ca-cert\") pod \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " Oct 30 02:08:29.461857 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.461826 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-workload-socket\") pod \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " Oct 30 02:08:29.461972 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.461860 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-data" pod "6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" (UID: "6aca41c6-d766-49a1-8b29-6e55cd7cb8bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:29.461972 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.461871 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-envoy\") pod \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " Oct 30 02:08:29.461972 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.461953 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-credential-socket\") pod \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\" (UID: \"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc\") " Oct 30 02:08:29.462175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.461998 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-workload-certs" pod "6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" (UID: "6aca41c6-d766-49a1-8b29-6e55cd7cb8bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:29.462233 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.462213 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-workload-socket" pod "6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" (UID: "6aca41c6-d766-49a1-8b29-6e55cd7cb8bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:29.462286 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.462226 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istiod-ca-cert" pod "6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" (UID: "6aca41c6-d766-49a1-8b29-6e55cd7cb8bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:08:29.462351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.462310 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-credential-socket" pod "6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" (UID: "6aca41c6-d766-49a1-8b29-6e55cd7cb8bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:29.462692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.462669 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:29.462692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.462691 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:29.462852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.462704 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:29.462852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.462716 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:29.462852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.462727 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:29.464421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.464395 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-token" pod "6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" (UID: "6aca41c6-d766-49a1-8b29-6e55cd7cb8bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:29.464544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.464473 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-kube-api-access-68wgw" pod "6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" (UID: "6aca41c6-d766-49a1-8b29-6e55cd7cb8bc"). InnerVolumeSpecName "kube-api-access-68wgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:29.464544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.464517 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-envoy" pod "6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" (UID: "6aca41c6-d766-49a1-8b29-6e55cd7cb8bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:29.464544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.464532 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-podinfo" pod "6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" (UID: "6aca41c6-d766-49a1-8b29-6e55cd7cb8bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:08:29.563873 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.563833 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:29.563873 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.563867 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:29.563873 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.563881 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68wgw\" (UniqueName: \"kubernetes.io/projected/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-kube-api-access-68wgw\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:29.564197 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.563895 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:29.985324 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.985289 5998 generic.go:356] "Generic (PLEG): container finished" podID="6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" containerID="3a734fa25185c308dc92d15f47078ab76d4e137255a5dea9af10a5b83172e6b2" exitCode=0 Oct 30 02:08:29.985510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.985337 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" Oct 30 02:08:29.985510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.985367 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" event={"ID":"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc","Type":"ContainerDied","Data":"3a734fa25185c308dc92d15f47078ab76d4e137255a5dea9af10a5b83172e6b2"} Oct 30 02:08:29.985510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.985426 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" event={"ID":"6aca41c6-d766-49a1-8b29-6e55cd7cb8bc","Type":"ContainerDied","Data":"38bc9859dbc2ee589fe27e13e6eae017858a87116c551afcee5f17edc3f68522"} Oct 30 02:08:29.985510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.985449 5998 scope.go:122] "RemoveContainer" containerID="3a734fa25185c308dc92d15f47078ab76d4e137255a5dea9af10a5b83172e6b2" Oct 30 02:08:29.998336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.998306 5998 scope.go:122] "RemoveContainer" containerID="3a734fa25185c308dc92d15f47078ab76d4e137255a5dea9af10a5b83172e6b2" Oct 30 02:08:29.998660 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:08:29.998642 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a734fa25185c308dc92d15f47078ab76d4e137255a5dea9af10a5b83172e6b2\": container with ID starting with 3a734fa25185c308dc92d15f47078ab76d4e137255a5dea9af10a5b83172e6b2 not found: ID does not exist" containerID="3a734fa25185c308dc92d15f47078ab76d4e137255a5dea9af10a5b83172e6b2" Oct 30 02:08:29.998730 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:29.998671 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a734fa25185c308dc92d15f47078ab76d4e137255a5dea9af10a5b83172e6b2"} err="failed to get container status \"3a734fa25185c308dc92d15f47078ab76d4e137255a5dea9af10a5b83172e6b2\": rpc error: code = NotFound desc = could not find container \"3a734fa25185c308dc92d15f47078ab76d4e137255a5dea9af10a5b83172e6b2\": container with ID starting with 3a734fa25185c308dc92d15f47078ab76d4e137255a5dea9af10a5b83172e6b2 not found: ID does not exist" Oct 30 02:08:30.045079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.045042 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt"] Oct 30 02:08:30.054682 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.054653 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt"] Oct 30 02:08:30.228334 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.228292 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:08:30.228520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.228350 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-2t5kt" podUID="6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:08:30.276271 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.276174 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:30.454780 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.454745 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm"] Oct 30 02:08:30.455321 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.455305 5998 cpu_manager.go:414] "containerMap: removing container" podUID="6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" containerName="istio-proxy" Oct 30 02:08:30.455371 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.455321 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" containerName="istio-proxy" Oct 30 02:08:30.455371 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.455361 5998 cpu_manager.go:414] "containerMap: removing container" podUID="0ad87fa4-78c9-4e60-8706-a78dbd1002b1" containerName="istio-proxy" Oct 30 02:08:30.455371 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.455366 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad87fa4-78c9-4e60-8706-a78dbd1002b1" containerName="istio-proxy" Oct 30 02:08:30.455474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.455377 5998 cpu_manager.go:414] "containerMap: removing container" podUID="0ad87fa4-78c9-4e60-8706-a78dbd1002b1" containerName="istio-proxy" Oct 30 02:08:30.455474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.455382 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad87fa4-78c9-4e60-8706-a78dbd1002b1" containerName="istio-proxy" Oct 30 02:08:30.455474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.455457 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad87fa4-78c9-4e60-8706-a78dbd1002b1" containerName="istio-proxy" Oct 30 02:08:30.455474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.455466 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad87fa4-78c9-4e60-8706-a78dbd1002b1" containerName="istio-proxy" Oct 30 02:08:30.455474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.455475 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" containerName="istio-proxy" Oct 30 02:08:30.460294 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.460268 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm" Oct 30 02:08:30.474923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.474901 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:08:30.475310 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.475291 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:08:30.475310 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.475289 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:08:30.475421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.475292 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:08:30.479956 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.479931 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm"] Oct 30 02:08:30.541440 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.541356 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aca41c6-d766-49a1-8b29-6e55cd7cb8bc" path="/var/lib/kubelet/pods/6aca41c6-d766-49a1-8b29-6e55cd7cb8bc/volumes" Oct 30 02:08:30.597577 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.597544 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a240736-76fb-44ae-a56a-f99f5e84b37c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e15a0e69-s98dm\" (UID: \"4a240736-76fb-44ae-a56a-f99f5e84b37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm" Oct 30 02:08:30.597731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.597656 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7sk5\" (UniqueName: \"kubernetes.io/projected/4a240736-76fb-44ae-a56a-f99f5e84b37c-kube-api-access-q7sk5\") pod \"maas-api-revoke-sub-e15a0e69-s98dm\" (UID: \"4a240736-76fb-44ae-a56a-f99f5e84b37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm" Oct 30 02:08:30.597731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.597714 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a240736-76fb-44ae-a56a-f99f5e84b37c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e15a0e69-s98dm\" (UID: \"4a240736-76fb-44ae-a56a-f99f5e84b37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm" Oct 30 02:08:30.699049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.699010 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7sk5\" (UniqueName: \"kubernetes.io/projected/4a240736-76fb-44ae-a56a-f99f5e84b37c-kube-api-access-q7sk5\") pod \"maas-api-revoke-sub-e15a0e69-s98dm\" (UID: \"4a240736-76fb-44ae-a56a-f99f5e84b37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm" Oct 30 02:08:30.699269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.699121 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a240736-76fb-44ae-a56a-f99f5e84b37c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e15a0e69-s98dm\" (UID: \"4a240736-76fb-44ae-a56a-f99f5e84b37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm" Oct 30 02:08:30.699431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.699403 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a240736-76fb-44ae-a56a-f99f5e84b37c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e15a0e69-s98dm\" (UID: \"4a240736-76fb-44ae-a56a-f99f5e84b37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm" Oct 30 02:08:30.711422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.711392 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:08:30.711604 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.711586 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:08:30.720201 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.720177 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a240736-76fb-44ae-a56a-f99f5e84b37c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e15a0e69-s98dm\" (UID: \"4a240736-76fb-44ae-a56a-f99f5e84b37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm" Oct 30 02:08:30.721819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.721799 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a240736-76fb-44ae-a56a-f99f5e84b37c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e15a0e69-s98dm\" (UID: \"4a240736-76fb-44ae-a56a-f99f5e84b37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm" Oct 30 02:08:30.730418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.730399 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:08:30.741842 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.741818 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7sk5\" (UniqueName: \"kubernetes.io/projected/4a240736-76fb-44ae-a56a-f99f5e84b37c-kube-api-access-q7sk5\") pod \"maas-api-revoke-sub-e15a0e69-s98dm\" (UID: \"4a240736-76fb-44ae-a56a-f99f5e84b37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm" Oct 30 02:08:30.785831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.785795 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:08:30.794045 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:30.793976 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm" Oct 30 02:08:31.108021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:31.107986 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm"] Oct 30 02:08:31.108211 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:08:31.108179 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a240736_76fb_44ae_a56a_f99f5e84b37c.slice/crio-4295bc70bbaac49db01c502ca7d9f92f9578586a2b038ce058f5c6d4a7aef005 WatchSource:0}: Error finding container 4295bc70bbaac49db01c502ca7d9f92f9578586a2b038ce058f5c6d4a7aef005: Status 404 returned error can't find the container with id 4295bc70bbaac49db01c502ca7d9f92f9578586a2b038ce058f5c6d4a7aef005 Oct 30 02:08:31.977683 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:31.977648 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:32.008614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.007764 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm" event={"ID":"4a240736-76fb-44ae-a56a-f99f5e84b37c","Type":"ContainerStarted","Data":"78db3a829cb93f277f8aa2f69dbe4096c421154b7ba9e412f5e23adb1b4eebd8"} Oct 30 02:08:32.008614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.007864 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm" event={"ID":"4a240736-76fb-44ae-a56a-f99f5e84b37c","Type":"ContainerStarted","Data":"4295bc70bbaac49db01c502ca7d9f92f9578586a2b038ce058f5c6d4a7aef005"} Oct 30 02:08:32.059666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.059601 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm" podStartSLOduration=2.059578882 podStartE2EDuration="2.059578882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:32.055676995 +0000 UTC m=+2341.634468069" watchObservedRunningTime="2026-10-30 02:08:32.059578882 +0000 UTC m=+2341.638369964" Oct 30 02:08:32.423264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.423232 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:32.843581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.843555 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:08:32.933648 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.933604 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24b4bd4-4eb0-412c-a54c-250a72795d94-kube-api-access-prbt9\" (UniqueName: \"kubernetes.io/projected/b24b4bd4-4eb0-412c-a54c-250a72795d94-kube-api-access-prbt9\") pod \"b24b4bd4-4eb0-412c-a54c-250a72795d94\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " Oct 30 02:08:32.933833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.933687 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-data\") pod \"b24b4bd4-4eb0-412c-a54c-250a72795d94\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " Oct 30 02:08:32.933833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.933741 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-podinfo\") pod \"b24b4bd4-4eb0-412c-a54c-250a72795d94\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " Oct 30 02:08:32.933833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.933777 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-envoy\") pod \"b24b4bd4-4eb0-412c-a54c-250a72795d94\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " Oct 30 02:08:32.933833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.933821 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-token\" (UniqueName: \"kubernetes.io/projected/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-token\") pod \"b24b4bd4-4eb0-412c-a54c-250a72795d94\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " Oct 30 02:08:32.934083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.933919 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-credential-socket\") pod \"b24b4bd4-4eb0-412c-a54c-250a72795d94\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " Oct 30 02:08:32.934083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.934065 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-workload-socket\") pod \"b24b4bd4-4eb0-412c-a54c-250a72795d94\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " Oct 30 02:08:32.934219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.934118 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-workload-certs\") pod \"b24b4bd4-4eb0-412c-a54c-250a72795d94\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " Oct 30 02:08:32.934272 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.934249 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-data" pod "b24b4bd4-4eb0-412c-a54c-250a72795d94" (UID: "b24b4bd4-4eb0-412c-a54c-250a72795d94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:32.934336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.934263 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24b4bd4-4eb0-412c-a54c-250a72795d94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24b4bd4-4eb0-412c-a54c-250a72795d94-istiod-ca-cert\") pod \"b24b4bd4-4eb0-412c-a54c-250a72795d94\" (UID: \"b24b4bd4-4eb0-412c-a54c-250a72795d94\") " Oct 30 02:08:32.934467 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.934396 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-credential-socket" pod "b24b4bd4-4eb0-412c-a54c-250a72795d94" (UID: "b24b4bd4-4eb0-412c-a54c-250a72795d94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:32.934538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.934470 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-workload-socket" pod "b24b4bd4-4eb0-412c-a54c-250a72795d94" (UID: "b24b4bd4-4eb0-412c-a54c-250a72795d94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:32.934588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.934560 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24b4bd4-4eb0-412c-a54c-250a72795d94-istiod-ca-cert" pod "b24b4bd4-4eb0-412c-a54c-250a72795d94" (UID: "b24b4bd4-4eb0-412c-a54c-250a72795d94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:08:32.934676 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.934655 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-workload-certs" pod "b24b4bd4-4eb0-412c-a54c-250a72795d94" (UID: "b24b4bd4-4eb0-412c-a54c-250a72795d94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:32.934992 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.934971 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:32.935075 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.934998 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:32.935075 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.935014 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:32.935075 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.935028 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:32.935075 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.935040 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24b4bd4-4eb0-412c-a54c-250a72795d94-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:32.943240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.939443 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-envoy" pod "b24b4bd4-4eb0-412c-a54c-250a72795d94" (UID: "b24b4bd4-4eb0-412c-a54c-250a72795d94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:08:32.943240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.939479 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24b4bd4-4eb0-412c-a54c-250a72795d94-kube-api-access-prbt9" pod "b24b4bd4-4eb0-412c-a54c-250a72795d94" (UID: "b24b4bd4-4eb0-412c-a54c-250a72795d94"). InnerVolumeSpecName "kube-api-access-prbt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:32.943240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.939504 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-token" pod "b24b4bd4-4eb0-412c-a54c-250a72795d94" (UID: "b24b4bd4-4eb0-412c-a54c-250a72795d94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:32.943240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:32.941285 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-podinfo" pod "b24b4bd4-4eb0-412c-a54c-250a72795d94" (UID: "b24b4bd4-4eb0-412c-a54c-250a72795d94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:08:33.022172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.022125 5998 generic.go:356] "Generic (PLEG): container finished" podID="b24b4bd4-4eb0-412c-a54c-250a72795d94" containerID="065632ed5247eef5acd54bcd4df4e5ad60b7fcccb360d758a5adf03310af8ff8" exitCode=0 Oct 30 02:08:33.022172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.022159 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" Oct 30 02:08:33.022662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.022186 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" event={"ID":"b24b4bd4-4eb0-412c-a54c-250a72795d94","Type":"ContainerDied","Data":"065632ed5247eef5acd54bcd4df4e5ad60b7fcccb360d758a5adf03310af8ff8"} Oct 30 02:08:33.022662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.022238 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" event={"ID":"b24b4bd4-4eb0-412c-a54c-250a72795d94","Type":"ContainerDied","Data":"9b16f0c9956d6c1fd187b6dfbdc2a94d01b0d95d503b3217e7d44fab04101f4a"} Oct 30 02:08:33.022662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.022262 5998 scope.go:122] "RemoveContainer" containerID="065632ed5247eef5acd54bcd4df4e5ad60b7fcccb360d758a5adf03310af8ff8" Oct 30 02:08:33.024042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.024018 5998 generic.go:356] "Generic (PLEG): container finished" podID="4a240736-76fb-44ae-a56a-f99f5e84b37c" containerID="78db3a829cb93f277f8aa2f69dbe4096c421154b7ba9e412f5e23adb1b4eebd8" exitCode=0 Oct 30 02:08:33.024202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.024145 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm" event={"ID":"4a240736-76fb-44ae-a56a-f99f5e84b37c","Type":"ContainerDied","Data":"78db3a829cb93f277f8aa2f69dbe4096c421154b7ba9e412f5e23adb1b4eebd8"} Oct 30 02:08:33.033391 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.033370 5998 scope.go:122] "RemoveContainer" containerID="c5ff79cac9535ecffdca44eb0a19caf2fc937f93bdb78b51ea57a2b28f69cac0" Oct 30 02:08:33.036058 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.036035 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prbt9\" (UniqueName: \"kubernetes.io/projected/b24b4bd4-4eb0-412c-a54c-250a72795d94-kube-api-access-prbt9\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:33.036194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.036063 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:33.036194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.036078 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:33.036194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.036105 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24b4bd4-4eb0-412c-a54c-250a72795d94-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:33.042864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.042847 5998 scope.go:122] "RemoveContainer" containerID="065632ed5247eef5acd54bcd4df4e5ad60b7fcccb360d758a5adf03310af8ff8" Oct 30 02:08:33.043148 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:08:33.043130 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"065632ed5247eef5acd54bcd4df4e5ad60b7fcccb360d758a5adf03310af8ff8\": container with ID starting with 065632ed5247eef5acd54bcd4df4e5ad60b7fcccb360d758a5adf03310af8ff8 not found: ID does not exist" containerID="065632ed5247eef5acd54bcd4df4e5ad60b7fcccb360d758a5adf03310af8ff8" Oct 30 02:08:33.043219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.043156 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"065632ed5247eef5acd54bcd4df4e5ad60b7fcccb360d758a5adf03310af8ff8"} err="failed to get container status \"065632ed5247eef5acd54bcd4df4e5ad60b7fcccb360d758a5adf03310af8ff8\": rpc error: code = NotFound desc = could not find container \"065632ed5247eef5acd54bcd4df4e5ad60b7fcccb360d758a5adf03310af8ff8\": container with ID starting with 065632ed5247eef5acd54bcd4df4e5ad60b7fcccb360d758a5adf03310af8ff8 not found: ID does not exist" Oct 30 02:08:33.043219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.043172 5998 scope.go:122] "RemoveContainer" containerID="c5ff79cac9535ecffdca44eb0a19caf2fc937f93bdb78b51ea57a2b28f69cac0" Oct 30 02:08:33.043447 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:08:33.043429 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5ff79cac9535ecffdca44eb0a19caf2fc937f93bdb78b51ea57a2b28f69cac0\": container with ID starting with c5ff79cac9535ecffdca44eb0a19caf2fc937f93bdb78b51ea57a2b28f69cac0 not found: ID does not exist" containerID="c5ff79cac9535ecffdca44eb0a19caf2fc937f93bdb78b51ea57a2b28f69cac0" Oct 30 02:08:33.043490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.043451 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5ff79cac9535ecffdca44eb0a19caf2fc937f93bdb78b51ea57a2b28f69cac0"} err="failed to get container status \"c5ff79cac9535ecffdca44eb0a19caf2fc937f93bdb78b51ea57a2b28f69cac0\": rpc error: code = NotFound desc = could not find container \"c5ff79cac9535ecffdca44eb0a19caf2fc937f93bdb78b51ea57a2b28f69cac0\": container with ID starting with c5ff79cac9535ecffdca44eb0a19caf2fc937f93bdb78b51ea57a2b28f69cac0 not found: ID does not exist" Oct 30 02:08:33.112876 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.112844 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s"] Oct 30 02:08:33.119253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.119223 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s"] Oct 30 02:08:33.673322 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.673289 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:33.805648 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.805610 5998 patch_prober.go:29] interesting pod/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:08:33.805837 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:33.805686 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-3ad281-openshift-default-9d85cfb97-vjd8s" podUID="b24b4bd4-4eb0-412c-a54c-250a72795d94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:08:34.209199 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.209159 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm" Oct 30 02:08:34.351777 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.351685 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a240736-76fb-44ae-a56a-f99f5e84b37c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a240736-76fb-44ae-a56a-f99f5e84b37c-maas-api-openshift-ca\") pod \"4a240736-76fb-44ae-a56a-f99f5e84b37c\" (UID: \"4a240736-76fb-44ae-a56a-f99f5e84b37c\") " Oct 30 02:08:34.351944 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.351819 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a240736-76fb-44ae-a56a-f99f5e84b37c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a240736-76fb-44ae-a56a-f99f5e84b37c-maas-api-xks-ca\") pod \"4a240736-76fb-44ae-a56a-f99f5e84b37c\" (UID: \"4a240736-76fb-44ae-a56a-f99f5e84b37c\") " Oct 30 02:08:34.351944 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.351871 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a240736-76fb-44ae-a56a-f99f5e84b37c-kube-api-access-q7sk5\" (UniqueName: \"kubernetes.io/projected/4a240736-76fb-44ae-a56a-f99f5e84b37c-kube-api-access-q7sk5\") pod \"4a240736-76fb-44ae-a56a-f99f5e84b37c\" (UID: \"4a240736-76fb-44ae-a56a-f99f5e84b37c\") " Oct 30 02:08:34.352234 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.352202 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a240736-76fb-44ae-a56a-f99f5e84b37c-maas-api-openshift-ca" pod "4a240736-76fb-44ae-a56a-f99f5e84b37c" (UID: "4a240736-76fb-44ae-a56a-f99f5e84b37c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:08:34.352463 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.352435 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a240736-76fb-44ae-a56a-f99f5e84b37c-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:34.354177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.354139 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a240736-76fb-44ae-a56a-f99f5e84b37c-maas-api-xks-ca" pod "4a240736-76fb-44ae-a56a-f99f5e84b37c" (UID: "4a240736-76fb-44ae-a56a-f99f5e84b37c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:08:34.354409 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.354386 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a240736-76fb-44ae-a56a-f99f5e84b37c-kube-api-access-q7sk5" pod "4a240736-76fb-44ae-a56a-f99f5e84b37c" (UID: "4a240736-76fb-44ae-a56a-f99f5e84b37c"). InnerVolumeSpecName "kube-api-access-q7sk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:34.453297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.453258 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a240736-76fb-44ae-a56a-f99f5e84b37c-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:34.453297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.453288 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7sk5\" (UniqueName: \"kubernetes.io/projected/4a240736-76fb-44ae-a56a-f99f5e84b37c-kube-api-access-q7sk5\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:34.540996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.540953 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b24b4bd4-4eb0-412c-a54c-250a72795d94" path="/var/lib/kubelet/pods/b24b4bd4-4eb0-412c-a54c-250a72795d94/volumes" Oct 30 02:08:34.683805 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.683762 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96"] Oct 30 02:08:34.684871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.684841 5998 cpu_manager.go:414] "containerMap: removing container" podUID="b24b4bd4-4eb0-412c-a54c-250a72795d94" containerName="istio-proxy" Oct 30 02:08:34.684871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.684869 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24b4bd4-4eb0-412c-a54c-250a72795d94" containerName="istio-proxy" Oct 30 02:08:34.685170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.684917 5998 cpu_manager.go:414] "containerMap: removing container" podUID="4a240736-76fb-44ae-a56a-f99f5e84b37c" containerName="revoke-keys" Oct 30 02:08:34.685170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.684939 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a240736-76fb-44ae-a56a-f99f5e84b37c" containerName="revoke-keys" Oct 30 02:08:34.685170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.684990 5998 cpu_manager.go:414] "containerMap: removing container" podUID="b24b4bd4-4eb0-412c-a54c-250a72795d94" containerName="istio-proxy" Oct 30 02:08:34.685170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.684999 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24b4bd4-4eb0-412c-a54c-250a72795d94" containerName="istio-proxy" Oct 30 02:08:34.685372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.685326 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24b4bd4-4eb0-412c-a54c-250a72795d94" containerName="istio-proxy" Oct 30 02:08:34.685372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.685349 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a240736-76fb-44ae-a56a-f99f5e84b37c" containerName="revoke-keys" Oct 30 02:08:34.685525 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.685374 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24b4bd4-4eb0-412c-a54c-250a72795d94" containerName="istio-proxy" Oct 30 02:08:34.690931 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.690905 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:34.691946 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.691925 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96"] Oct 30 02:08:34.704693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.704661 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-4153e5\"/\"kube-root-ca.crt\"" Oct 30 02:08:34.704693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.704663 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-4153e5\"/\"openshift-service-ca.crt\"" Oct 30 02:08:34.704845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.704767 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-4153e5\"/\"default-dockercfg-gj4h2\"" Oct 30 02:08:34.714996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.714966 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-4153e5\"/\"auth-test-model-4153e5-kserve-self-signed-certs\"" Oct 30 02:08:34.716978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.716945 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:34.859323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.859276 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f826281b-5ba7-427b-800b-e8fbb621793a-tls-certs\") pod \"auth-test-model-4153e5-kserve-748754bc97-nzs96\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:34.859323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.859326 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-home\") pod \"auth-test-model-4153e5-kserve-748754bc97-nzs96\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:34.859602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.859361 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-model-cache\") pod \"auth-test-model-4153e5-kserve-748754bc97-nzs96\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:34.859602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.859521 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-dshm\") pod \"auth-test-model-4153e5-kserve-748754bc97-nzs96\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:34.859602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.859593 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd8f5\" (UniqueName: \"kubernetes.io/projected/f826281b-5ba7-427b-800b-e8fbb621793a-kube-api-access-kd8f5\") pod \"auth-test-model-4153e5-kserve-748754bc97-nzs96\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:34.859761 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.859648 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-tmp-dir\") pod \"auth-test-model-4153e5-kserve-748754bc97-nzs96\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:34.963175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.963002 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f826281b-5ba7-427b-800b-e8fbb621793a-tls-certs\") pod \"auth-test-model-4153e5-kserve-748754bc97-nzs96\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:34.963175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.963077 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-home\") pod \"auth-test-model-4153e5-kserve-748754bc97-nzs96\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:34.963175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.963138 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-model-cache\") pod \"auth-test-model-4153e5-kserve-748754bc97-nzs96\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:34.963470 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.963213 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-dshm\") pod \"auth-test-model-4153e5-kserve-748754bc97-nzs96\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:34.963470 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.963275 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd8f5\" (UniqueName: \"kubernetes.io/projected/f826281b-5ba7-427b-800b-e8fbb621793a-kube-api-access-kd8f5\") pod \"auth-test-model-4153e5-kserve-748754bc97-nzs96\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:34.963470 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.963325 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-tmp-dir\") pod \"auth-test-model-4153e5-kserve-748754bc97-nzs96\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:34.968244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.963852 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-tmp-dir\") pod \"auth-test-model-4153e5-kserve-748754bc97-nzs96\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:34.968244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.964084 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-home\") pod \"auth-test-model-4153e5-kserve-748754bc97-nzs96\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:34.968244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.964359 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-model-cache\") pod \"auth-test-model-4153e5-kserve-748754bc97-nzs96\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:34.968244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.967398 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-dshm\") pod \"auth-test-model-4153e5-kserve-748754bc97-nzs96\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:34.976490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.976130 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f826281b-5ba7-427b-800b-e8fbb621793a-tls-certs\") pod \"auth-test-model-4153e5-kserve-748754bc97-nzs96\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:34.987022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:34.986991 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd8f5\" (UniqueName: \"kubernetes.io/projected/f826281b-5ba7-427b-800b-e8fbb621793a-kube-api-access-kd8f5\") pod \"auth-test-model-4153e5-kserve-748754bc97-nzs96\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:35.006015 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:35.005946 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:35.050302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:35.050275 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm" Oct 30 02:08:35.050460 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:35.050426 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm" event={"ID":"4a240736-76fb-44ae-a56a-f99f5e84b37c","Type":"ContainerDied","Data":"4295bc70bbaac49db01c502ca7d9f92f9578586a2b038ce058f5c6d4a7aef005"} Oct 30 02:08:35.050570 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:35.050557 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4295bc70bbaac49db01c502ca7d9f92f9578586a2b038ce058f5c6d4a7aef005" Oct 30 02:08:35.302910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:35.302809 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:35.372421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:35.372346 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96"] Oct 30 02:08:35.384207 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:08:35.384169 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf826281b_5ba7_427b_800b_e8fbb621793a.slice/crio-2fe24ed4388eb88fbf87463de271207657d26f20fbac888b273fb46d291f7b8f WatchSource:0}: Error finding container 2fe24ed4388eb88fbf87463de271207657d26f20fbac888b273fb46d291f7b8f: Status 404 returned error can't find the container with id 2fe24ed4388eb88fbf87463de271207657d26f20fbac888b273fb46d291f7b8f Oct 30 02:08:35.836646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:35.836612 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:35.959131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:35.959057 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:36.060487 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:36.060447 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" event={"ID":"f826281b-5ba7-427b-800b-e8fbb621793a","Type":"ContainerStarted","Data":"864321e8d55cb7cf36aabf8ca66cb9ea3137c595c305dcc9574f2242df6c967e"} Oct 30 02:08:36.060487 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:36.060501 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" event={"ID":"f826281b-5ba7-427b-800b-e8fbb621793a","Type":"ContainerStarted","Data":"2fe24ed4388eb88fbf87463de271207657d26f20fbac888b273fb46d291f7b8f"} Oct 30 02:08:36.102245 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:36.102140 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" podStartSLOduration=2.102123364 podStartE2EDuration="2.102123364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:36.098728832 +0000 UTC m=+2345.677519915" watchObservedRunningTime="2026-10-30 02:08:36.102123364 +0000 UTC m=+2345.680914445" Oct 30 02:08:36.522107 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:36.522061 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:36.637486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:36.637458 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:36.911159 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:36.911123 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:37.007220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:37.007181 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:37.350912 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:37.350810 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:39.142334 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:39.142293 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:40.486742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:40.486707 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:40.981795 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:40.981760 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:41.120429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:41.120394 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:41.255877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:41.255771 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:41.942165 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:41.941470 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:42.504023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.503984 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str"] Oct 30 02:08:42.509310 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.509284 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.532312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.532250 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-584ec5-openshift-default-dockercfg-7kbq5\"" Oct 30 02:08:42.545840 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.545801 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str"] Oct 30 02:08:42.652118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.644003 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-token\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.652118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.644194 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-podinfo\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.652118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.644235 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-credential-socket\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.652118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.644263 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-workload-certs\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.652118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.644338 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-data\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.652118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.644470 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kph7q\" (UniqueName: \"kubernetes.io/projected/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-kube-api-access-kph7q\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.652118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.644568 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-envoy\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.652118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.644603 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istiod-ca-cert\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.652118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.644648 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-workload-socket\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.678157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.656601 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:42.779186 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.779071 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kph7q\" (UniqueName: \"kubernetes.io/projected/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-kube-api-access-kph7q\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.780430 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.779886 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-envoy\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.780663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.780639 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istiod-ca-cert\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.781938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.781918 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-workload-socket\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.782060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.782011 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-token\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.782171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.781853 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istiod-ca-cert\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.782240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.782166 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-podinfo\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.782240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.782203 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-credential-socket\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.782240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.782228 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-workload-certs\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.782422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.782310 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-data\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.783210 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.782490 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-workload-socket\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.783210 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.782712 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-data\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.783400 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.783348 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-credential-socket\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.783400 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.783391 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-workload-certs\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.787028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.786951 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-envoy\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.789760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.789600 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-podinfo\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.812738 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.812672 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-token\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.813673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.813630 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kph7q\" (UniqueName: \"kubernetes.io/projected/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-kube-api-access-kph7q\") pod \"e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.835538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.835499 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:42.863945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:42.852525 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:43.221635 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:43.221569 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str"] Oct 30 02:08:43.237898 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:08:43.237861 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ec48a4_d344_4e9c_bba3_b1b68bc21a84.slice/crio-a41dc79e82a01b9ebd8ed5b1c63a3fccec3b30603612279083096faa1e0d20e5 WatchSource:0}: Error finding container a41dc79e82a01b9ebd8ed5b1c63a3fccec3b30603612279083096faa1e0d20e5: Status 404 returned error can't find the container with id a41dc79e82a01b9ebd8ed5b1c63a3fccec3b30603612279083096faa1e0d20e5 Oct 30 02:08:43.247328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:43.245920 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:08:43.247328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:43.246009 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:08:43.247328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:43.246052 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:08:43.253621 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:43.253582 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:44.164967 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.164921 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" event={"ID":"29ec48a4-d344-4e9c-bba3-b1b68bc21a84","Type":"ContainerStarted","Data":"46d21d37f9becc1c0ef4e0a192c45cbe5dd8bfd2a9fb4829ac9d77ee63132ce5"} Oct 30 02:08:44.165221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.165028 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" event={"ID":"29ec48a4-d344-4e9c-bba3-b1b68bc21a84","Type":"ContainerStarted","Data":"a41dc79e82a01b9ebd8ed5b1c63a3fccec3b30603612279083096faa1e0d20e5"} Oct 30 02:08:44.223336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.223278 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" podStartSLOduration=2.22326252 podStartE2EDuration="2.22326252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:44.221515517 +0000 UTC m=+2353.800306593" watchObservedRunningTime="2026-10-30 02:08:44.22326252 +0000 UTC m=+2353.802053601" Oct 30 02:08:44.287805 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.287769 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l"] Oct 30 02:08:44.292075 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.292057 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l" Oct 30 02:08:44.304619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.304585 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56ff3412-ngq2l\" (UID: \"f0bfdf1b-a614-4293-911d-f2c8ce5be38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l" Oct 30 02:08:44.304758 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.304652 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sgrw\" (UniqueName: \"kubernetes.io/projected/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-kube-api-access-6sgrw\") pod \"maas-api-revoke-sub-56ff3412-ngq2l\" (UID: \"f0bfdf1b-a614-4293-911d-f2c8ce5be38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l" Oct 30 02:08:44.304805 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.304756 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56ff3412-ngq2l\" (UID: \"f0bfdf1b-a614-4293-911d-f2c8ce5be38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l" Oct 30 02:08:44.305402 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.305380 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:08:44.305664 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.305645 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:08:44.305664 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.305653 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:08:44.308182 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.308159 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:08:44.314506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.314481 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l"] Oct 30 02:08:44.416360 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.415551 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56ff3412-ngq2l\" (UID: \"f0bfdf1b-a614-4293-911d-f2c8ce5be38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l" Oct 30 02:08:44.416360 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.415650 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sgrw\" (UniqueName: \"kubernetes.io/projected/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-kube-api-access-6sgrw\") pod \"maas-api-revoke-sub-56ff3412-ngq2l\" (UID: \"f0bfdf1b-a614-4293-911d-f2c8ce5be38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l" Oct 30 02:08:44.416360 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.415688 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56ff3412-ngq2l\" (UID: \"f0bfdf1b-a614-4293-911d-f2c8ce5be38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l" Oct 30 02:08:44.418602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.418335 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56ff3412-ngq2l\" (UID: \"f0bfdf1b-a614-4293-911d-f2c8ce5be38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l" Oct 30 02:08:44.423140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.421009 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56ff3412-ngq2l\" (UID: \"f0bfdf1b-a614-4293-911d-f2c8ce5be38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l" Oct 30 02:08:44.443175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.440549 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sgrw\" (UniqueName: \"kubernetes.io/projected/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-kube-api-access-6sgrw\") pod \"maas-api-revoke-sub-56ff3412-ngq2l\" (UID: \"f0bfdf1b-a614-4293-911d-f2c8ce5be38e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l" Oct 30 02:08:44.606975 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.606933 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l" Oct 30 02:08:44.836349 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.836310 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:44.842306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.842276 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:44.934740 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:44.934704 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l"] Oct 30 02:08:44.934893 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:08:44.934715 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bfdf1b_a614_4293_911d_f2c8ce5be38e.slice/crio-c1d79577066da21faf21d25858b9ae54449b27bd7ff881fb51bb39d6b35cb4ba WatchSource:0}: Error finding container c1d79577066da21faf21d25858b9ae54449b27bd7ff881fb51bb39d6b35cb4ba: Status 404 returned error can't find the container with id c1d79577066da21faf21d25858b9ae54449b27bd7ff881fb51bb39d6b35cb4ba Oct 30 02:08:45.006689 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:45.006660 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:45.020283 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:45.020254 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:45.025030 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:45.025004 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:45.032913 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:45.032891 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:08:45.188163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:45.185285 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l" event={"ID":"f0bfdf1b-a614-4293-911d-f2c8ce5be38e","Type":"ContainerStarted","Data":"c1d79577066da21faf21d25858b9ae54449b27bd7ff881fb51bb39d6b35cb4ba"} Oct 30 02:08:45.188163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:45.185720 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:45.188163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:45.187520 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:08:45.203687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:45.202182 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:08:46.243602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:46.241220 5998 generic.go:356] "Generic (PLEG): container finished" podID="f0bfdf1b-a614-4293-911d-f2c8ce5be38e" containerID="8b47917fcd557c7c85013456921e4c1dcbce116d5eda88bc3a8957378978428a" exitCode=0 Oct 30 02:08:46.243602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:46.242687 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l" event={"ID":"f0bfdf1b-a614-4293-911d-f2c8ce5be38e","Type":"ContainerDied","Data":"8b47917fcd557c7c85013456921e4c1dcbce116d5eda88bc3a8957378978428a"} Oct 30 02:08:46.709730 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:46.709689 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:46.895669 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:46.895631 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:47.423123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:47.421928 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:47.993058 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:47.993029 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l" Oct 30 02:08:48.158156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:48.157470 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-kube-api-access-6sgrw\" (UniqueName: \"kubernetes.io/projected/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-kube-api-access-6sgrw\") pod \"f0bfdf1b-a614-4293-911d-f2c8ce5be38e\" (UID: \"f0bfdf1b-a614-4293-911d-f2c8ce5be38e\") " Oct 30 02:08:48.158156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:48.157595 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-maas-api-xks-ca\") pod \"f0bfdf1b-a614-4293-911d-f2c8ce5be38e\" (UID: \"f0bfdf1b-a614-4293-911d-f2c8ce5be38e\") " Oct 30 02:08:48.158156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:48.157784 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-maas-api-openshift-ca\") pod \"f0bfdf1b-a614-4293-911d-f2c8ce5be38e\" (UID: \"f0bfdf1b-a614-4293-911d-f2c8ce5be38e\") " Oct 30 02:08:48.160711 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:48.160674 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-maas-api-openshift-ca" pod "f0bfdf1b-a614-4293-911d-f2c8ce5be38e" (UID: "f0bfdf1b-a614-4293-911d-f2c8ce5be38e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:08:48.163507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:48.163479 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-kube-api-access-6sgrw" pod "f0bfdf1b-a614-4293-911d-f2c8ce5be38e" (UID: "f0bfdf1b-a614-4293-911d-f2c8ce5be38e"). InnerVolumeSpecName "kube-api-access-6sgrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:08:48.171110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:48.170362 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-maas-api-xks-ca" pod "f0bfdf1b-a614-4293-911d-f2c8ce5be38e" (UID: "f0bfdf1b-a614-4293-911d-f2c8ce5be38e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:08:48.259155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:48.259118 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:48.259155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:48.259149 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sgrw\" (UniqueName: \"kubernetes.io/projected/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-kube-api-access-6sgrw\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:48.259155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:48.259161 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0bfdf1b-a614-4293-911d-f2c8ce5be38e-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:08:48.272750 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:48.272720 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l" Oct 30 02:08:48.272901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:48.272763 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l" event={"ID":"f0bfdf1b-a614-4293-911d-f2c8ce5be38e","Type":"ContainerDied","Data":"c1d79577066da21faf21d25858b9ae54449b27bd7ff881fb51bb39d6b35cb4ba"} Oct 30 02:08:48.272901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:48.272838 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1d79577066da21faf21d25858b9ae54449b27bd7ff881fb51bb39d6b35cb4ba" Oct 30 02:08:48.683141 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:48.683084 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:48.834737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:48.834691 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:49.840121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:49.840053 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:51.595115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:51.595068 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:52.151353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:52.151321 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:53.272393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:53.272351 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:54.297266 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:54.297227 5998 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:08:54.297754 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:54.297305 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" podUID="5220aca0-19d7-444c-b92b-da5361f306dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:08:54.911899 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:54.911862 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:55.222499 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:08:55.222392 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:08:55.742724 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:55.742693 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:56.189598 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.189560 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5"] Oct 30 02:08:56.190751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.190722 5998 cpu_manager.go:414] "containerMap: removing container" podUID="f0bfdf1b-a614-4293-911d-f2c8ce5be38e" containerName="revoke-keys" Oct 30 02:08:56.190933 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.190918 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0bfdf1b-a614-4293-911d-f2c8ce5be38e" containerName="revoke-keys" Oct 30 02:08:56.191215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.191200 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0bfdf1b-a614-4293-911d-f2c8ce5be38e" containerName="revoke-keys" Oct 30 02:08:56.198276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.198250 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.214762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.214118 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-a1028d-openshift-default-dockercfg-6l6qt\"" Oct 30 02:08:56.237986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.235143 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5"] Oct 30 02:08:56.283664 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.283621 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:56.317582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.317541 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-envoy\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.317766 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.317589 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-podinfo\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.317766 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.317720 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-data\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.317766 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.317763 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-workload-certs\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.317924 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.317821 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-credential-socket\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.317924 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.317890 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rst7c\" (UniqueName: \"kubernetes.io/projected/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-kube-api-access-rst7c\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.318034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.317933 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-token\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.318128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.318078 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istiod-ca-cert\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.318200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.318145 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-workload-socket\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.448770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.448664 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istiod-ca-cert\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.449174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.449147 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-workload-socket\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.466983 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.466949 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-envoy\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.467283 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.467262 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-podinfo\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.467511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.467493 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-data\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.468071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.468046 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-workload-certs\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.468289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.468271 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-credential-socket\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.468450 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.468435 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rst7c\" (UniqueName: \"kubernetes.io/projected/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-kube-api-access-rst7c\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.468613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.468599 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-token\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.470199 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.456437 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-workload-socket\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.471518 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.471489 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-workload-certs\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.471820 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.471793 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-data\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.472154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.472121 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-credential-socket\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.472247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.449927 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istiod-ca-cert\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.521040 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.517125 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-token\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.535194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.530494 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rst7c\" (UniqueName: \"kubernetes.io/projected/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-kube-api-access-rst7c\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.574136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.572777 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:56.578138 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.576927 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-podinfo\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.578138 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.577597 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-envoy\") pod \"e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:56.623151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.619794 5998 generic.go:356] "Generic (PLEG): container finished" podID="5220aca0-19d7-444c-b92b-da5361f306dc" containerID="178b62369ec547602268c4a39da502ed1f1a1d42b6ffddfc1a60baacbbbcf2a2" exitCode=0 Oct 30 02:08:56.634511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.630529 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" event={"ID":"5220aca0-19d7-444c-b92b-da5361f306dc","Type":"ContainerDied","Data":"178b62369ec547602268c4a39da502ed1f1a1d42b6ffddfc1a60baacbbbcf2a2"} Oct 30 02:08:56.656493 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.656372 5998 scope.go:122] "RemoveContainer" containerID="178b62369ec547602268c4a39da502ed1f1a1d42b6ffddfc1a60baacbbbcf2a2" Oct 30 02:08:56.669442 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.669398 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:08:56.670104 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.670056 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:08:56.670215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.670160 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:08:56.751370 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.751336 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:56.845648 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:56.837356 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:57.219845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:57.219813 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5"] Oct 30 02:08:57.221642 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:08:57.221601 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcad96d8_5e9b_4083_bbb0_4533bfd7e813.slice/crio-8c597c68f4fde5f9dbcf4e20176a6dcfb7540c3f5095e51c9d7a5f0e8d6beeca WatchSource:0}: Error finding container 8c597c68f4fde5f9dbcf4e20176a6dcfb7540c3f5095e51c9d7a5f0e8d6beeca: Status 404 returned error can't find the container with id 8c597c68f4fde5f9dbcf4e20176a6dcfb7540c3f5095e51c9d7a5f0e8d6beeca Oct 30 02:08:57.227950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:57.227643 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:08:57.227950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:57.227736 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:08:57.227950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:57.227779 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:08:57.644023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:57.643932 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" event={"ID":"5220aca0-19d7-444c-b92b-da5361f306dc","Type":"ContainerStarted","Data":"4b68904613460380e5d659b652bfdf2853da1b1e6cac0b2703e7a0f4fa44269a"} Oct 30 02:08:57.646365 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:57.646333 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" event={"ID":"dcad96d8-5e9b-4083-bbb0-4533bfd7e813","Type":"ContainerStarted","Data":"5d13e8f6c074d33897484e0a2030b241276e138ff7c2519d20ae2332edab2cc9"} Oct 30 02:08:57.646520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:57.646392 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" event={"ID":"dcad96d8-5e9b-4083-bbb0-4533bfd7e813","Type":"ContainerStarted","Data":"8c597c68f4fde5f9dbcf4e20176a6dcfb7540c3f5095e51c9d7a5f0e8d6beeca"} Oct 30 02:08:57.737496 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:57.736438 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" podStartSLOduration=1.736417063 podStartE2EDuration="1.736417063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:57.732899326 +0000 UTC m=+2367.311690407" watchObservedRunningTime="2026-10-30 02:08:57.736417063 +0000 UTC m=+2367.315208147" Oct 30 02:08:57.781416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:57.781382 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:08:57.781867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:57.781591 5998 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 02:08:57.781867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:57.781648 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" podUID="5220aca0-19d7-444c-b92b-da5361f306dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 02:08:57.838454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:57.838402 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:58.496743 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:58.496519 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:58.781965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:58.781881 5998 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 02:08:58.781965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:58.781939 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" podUID="5220aca0-19d7-444c-b92b-da5361f306dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 02:08:58.844381 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:58.844344 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:59.047153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.047017 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb"] Oct 30 02:08:59.055654 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.055613 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb"] Oct 30 02:08:59.055807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.055721 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.071275 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.070550 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-2819d2\"/\"auth-test-model-2819d2-kserve-self-signed-certs\"" Oct 30 02:08:59.071275 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.070647 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-2819d2\"/\"default-dockercfg-kjwph\"" Oct 30 02:08:59.071505 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.071291 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-2819d2\"/\"openshift-service-ca.crt\"" Oct 30 02:08:59.071505 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.071376 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-2819d2\"/\"kube-root-ca.crt\"" Oct 30 02:08:59.091425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.081472 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:08:59.198590 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.198546 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-tmp-dir\") pod \"auth-test-model-2819d2-kserve-5fc754cb9c-nlclb\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.198847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.198738 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2cd2f87-e701-4574-8193-901a704f2c80-tls-certs\") pod \"auth-test-model-2819d2-kserve-5fc754cb9c-nlclb\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.199011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.198982 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-dshm\") pod \"auth-test-model-2819d2-kserve-5fc754cb9c-nlclb\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.199073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.199046 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rjvl\" (UniqueName: \"kubernetes.io/projected/a2cd2f87-e701-4574-8193-901a704f2c80-kube-api-access-7rjvl\") pod \"auth-test-model-2819d2-kserve-5fc754cb9c-nlclb\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.199164 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.199117 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-home\") pod \"auth-test-model-2819d2-kserve-5fc754cb9c-nlclb\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.199280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.199262 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-model-cache\") pod \"auth-test-model-2819d2-kserve-5fc754cb9c-nlclb\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.321193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.300333 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-dshm\") pod \"auth-test-model-2819d2-kserve-5fc754cb9c-nlclb\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.321193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.300406 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rjvl\" (UniqueName: \"kubernetes.io/projected/a2cd2f87-e701-4574-8193-901a704f2c80-kube-api-access-7rjvl\") pod \"auth-test-model-2819d2-kserve-5fc754cb9c-nlclb\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.321193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.300440 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-home\") pod \"auth-test-model-2819d2-kserve-5fc754cb9c-nlclb\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.321193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.300566 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-model-cache\") pod \"auth-test-model-2819d2-kserve-5fc754cb9c-nlclb\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.321193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.300652 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-tmp-dir\") pod \"auth-test-model-2819d2-kserve-5fc754cb9c-nlclb\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.321193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.300711 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2cd2f87-e701-4574-8193-901a704f2c80-tls-certs\") pod \"auth-test-model-2819d2-kserve-5fc754cb9c-nlclb\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.345115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.329718 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-home\") pod \"auth-test-model-2819d2-kserve-5fc754cb9c-nlclb\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.345115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.330479 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-tmp-dir\") pod \"auth-test-model-2819d2-kserve-5fc754cb9c-nlclb\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.345115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.330763 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-model-cache\") pod \"auth-test-model-2819d2-kserve-5fc754cb9c-nlclb\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.387115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.386213 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rjvl\" (UniqueName: \"kubernetes.io/projected/a2cd2f87-e701-4574-8193-901a704f2c80-kube-api-access-7rjvl\") pod \"auth-test-model-2819d2-kserve-5fc754cb9c-nlclb\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.434153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.423935 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2cd2f87-e701-4574-8193-901a704f2c80-tls-certs\") pod \"auth-test-model-2819d2-kserve-5fc754cb9c-nlclb\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.494114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.489304 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-dshm\") pod \"auth-test-model-2819d2-kserve-5fc754cb9c-nlclb\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.666761 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.666724 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:59.667797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.667769 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:08:59.672979 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.672959 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:08:59.785527 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.785488 5998 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 02:08:59.790833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:08:59.785553 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" podUID="5220aca0-19d7-444c-b92b-da5361f306dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 02:09:00.040076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:00.040035 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb"] Oct 30 02:09:00.042802 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:09:00.042763 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2cd2f87_e701_4574_8193_901a704f2c80.slice/crio-a1de1b136ecb18fa6a9cdb262381798f29f9344da065732cfeef1fd54c9506ed WatchSource:0}: Error finding container a1de1b136ecb18fa6a9cdb262381798f29f9344da065732cfeef1fd54c9506ed: Status 404 returned error can't find the container with id a1de1b136ecb18fa6a9cdb262381798f29f9344da065732cfeef1fd54c9506ed Oct 30 02:09:00.140794 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:00.140753 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:00.272157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:00.272072 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:00.683818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:00.683773 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" event={"ID":"a2cd2f87-e701-4574-8193-901a704f2c80","Type":"ContainerStarted","Data":"23cc555a8effafe828290b18b4c40d79d05d087abb664cb8ba6b6b4ee19ea542"} Oct 30 02:09:00.683818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:00.683819 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" event={"ID":"a2cd2f87-e701-4574-8193-901a704f2c80","Type":"ContainerStarted","Data":"a1de1b136ecb18fa6a9cdb262381798f29f9344da065732cfeef1fd54c9506ed"} Oct 30 02:09:00.724186 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:00.722241 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:00.781072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:00.781040 5998 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 02:09:00.781259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:00.781105 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" podUID="5220aca0-19d7-444c-b92b-da5361f306dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 02:09:01.673296 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:01.673265 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:09:01.781283 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:01.781247 5998 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 02:09:01.781430 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:01.781310 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" podUID="5220aca0-19d7-444c-b92b-da5361f306dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 02:09:01.972189 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:01.972077 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:02.091929 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:02.091895 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:02.781168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:02.781127 5998 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 02:09:02.781630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:02.781192 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" podUID="5220aca0-19d7-444c-b92b-da5361f306dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 02:09:02.784190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:02.784150 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:03.785961 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:03.785929 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:09:03.786455 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:03.786429 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:09:03.787186 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:03.787161 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:09:03.845769 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:03.845699 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" podStartSLOduration=5.845677488 podStartE2EDuration="5.845677488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:09:00.752152736 +0000 UTC m=+2370.330943819" watchObservedRunningTime="2026-10-30 02:09:03.845677488 +0000 UTC m=+2373.424468570" Oct 30 02:09:04.597816 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:04.597779 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:06.216248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:06.216214 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:07.015362 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:07.015323 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:08.046486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:08.046437 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:09.673251 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:09.673206 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:09:09.674057 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:09.674027 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:09.686270 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:09.686245 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:09:09.808945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:09.808913 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:09:09.815225 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:09.812833 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:11.913682 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:11.913636 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:12.262578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.262475 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b"] Oct 30 02:09:12.266970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.266943 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.311033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.310997 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-17e747\"/\"openshift-service-ca.crt\"" Oct 30 02:09:12.311297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.311264 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-17e747\"/\"default-dockercfg-rl2hv\"" Oct 30 02:09:12.311417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.311377 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-17e747\"/\"auth-test-model-17e747-kserve-self-signed-certs\"" Oct 30 02:09:12.311607 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.311401 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-17e747\"/\"kube-root-ca.crt\"" Oct 30 02:09:12.319397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.317946 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b"] Oct 30 02:09:12.331779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.331749 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-tmp-dir\") pod \"auth-test-model-17e747-kserve-6d7fc45597-s572b\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.331978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.331807 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f22wj\" (UniqueName: \"kubernetes.io/projected/ec8fbe57-dca0-459c-a46d-e734d53ea28a-kube-api-access-f22wj\") pod \"auth-test-model-17e747-kserve-6d7fc45597-s572b\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.331978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.331855 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-home\") pod \"auth-test-model-17e747-kserve-6d7fc45597-s572b\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.331978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.331884 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8fbe57-dca0-459c-a46d-e734d53ea28a-tls-certs\") pod \"auth-test-model-17e747-kserve-6d7fc45597-s572b\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.331978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.331912 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-model-cache\") pod \"auth-test-model-17e747-kserve-6d7fc45597-s572b\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.331978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.331936 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-dshm\") pod \"auth-test-model-17e747-kserve-6d7fc45597-s572b\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.350200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.350167 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:12.435104 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.435060 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8fbe57-dca0-459c-a46d-e734d53ea28a-tls-certs\") pod \"auth-test-model-17e747-kserve-6d7fc45597-s572b\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.435285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.435125 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-model-cache\") pod \"auth-test-model-17e747-kserve-6d7fc45597-s572b\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.435285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.435145 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-dshm\") pod \"auth-test-model-17e747-kserve-6d7fc45597-s572b\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.435377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.435336 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-tmp-dir\") pod \"auth-test-model-17e747-kserve-6d7fc45597-s572b\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.435377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.435369 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f22wj\" (UniqueName: \"kubernetes.io/projected/ec8fbe57-dca0-459c-a46d-e734d53ea28a-kube-api-access-f22wj\") pod \"auth-test-model-17e747-kserve-6d7fc45597-s572b\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.435463 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.435402 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-home\") pod \"auth-test-model-17e747-kserve-6d7fc45597-s572b\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.435764 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.435741 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-home\") pod \"auth-test-model-17e747-kserve-6d7fc45597-s572b\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.437948 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.436599 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-tmp-dir\") pod \"auth-test-model-17e747-kserve-6d7fc45597-s572b\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.437948 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.436627 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-model-cache\") pod \"auth-test-model-17e747-kserve-6d7fc45597-s572b\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.439382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.439353 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-dshm\") pod \"auth-test-model-17e747-kserve-6d7fc45597-s572b\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.439788 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.439768 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8fbe57-dca0-459c-a46d-e734d53ea28a-tls-certs\") pod \"auth-test-model-17e747-kserve-6d7fc45597-s572b\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.459119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.459076 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f22wj\" (UniqueName: \"kubernetes.io/projected/ec8fbe57-dca0-459c-a46d-e734d53ea28a-kube-api-access-f22wj\") pod \"auth-test-model-17e747-kserve-6d7fc45597-s572b\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:12.613168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:12.607478 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:13.015917 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:13.015886 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b"] Oct 30 02:09:13.017763 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:09:13.017731 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec8fbe57_dca0_459c_a46d_e734d53ea28a.slice/crio-65686d7caee6a172f1cabbcbaf80a06f0c504430132a100f3cdc30fa7907ffeb WatchSource:0}: Error finding container 65686d7caee6a172f1cabbcbaf80a06f0c504430132a100f3cdc30fa7907ffeb: Status 404 returned error can't find the container with id 65686d7caee6a172f1cabbcbaf80a06f0c504430132a100f3cdc30fa7907ffeb Oct 30 02:09:13.826968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:13.826924 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" event={"ID":"ec8fbe57-dca0-459c-a46d-e734d53ea28a","Type":"ContainerStarted","Data":"9bf3f0727bb7eda840655d244264a3ed284a6131251bef902548ba061b69bb1b"} Oct 30 02:09:13.827175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:13.826985 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" event={"ID":"ec8fbe57-dca0-459c-a46d-e734d53ea28a","Type":"ContainerStarted","Data":"65686d7caee6a172f1cabbcbaf80a06f0c504430132a100f3cdc30fa7907ffeb"} Oct 30 02:09:13.874370 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:13.874314 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" podStartSLOduration=1.874298375 podStartE2EDuration="1.874298375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:09:13.869969238 +0000 UTC m=+2383.448760330" watchObservedRunningTime="2026-10-30 02:09:13.874298375 +0000 UTC m=+2383.453089507" Oct 30 02:09:14.053608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:14.053572 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:14.610073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:14.610031 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:15.807592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:15.807551 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:17.628595 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:17.628458 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:19.303666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:19.303625 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:20.908818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:20.908780 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:22.609728 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:22.609685 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:22.622829 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:22.622803 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:22.685330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:22.685296 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:22.941520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:22.941488 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:09:24.457484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:24.457449 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:25.225960 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:25.225918 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:25.267062 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:09:25.267012 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:09:26.348565 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:26.348531 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:26.871873 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:26.871836 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np"] Oct 30 02:09:26.876904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:26.876878 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:26.899819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:26.899788 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np"] Oct 30 02:09:26.904630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:26.904587 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-24fb97\"/\"kube-root-ca.crt\"" Oct 30 02:09:26.904884 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:26.904587 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-24fb97\"/\"openshift-service-ca.crt\"" Oct 30 02:09:26.904884 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:26.904741 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-24fb97\"/\"auth-test-model-24fb97-kserve-self-signed-certs\"" Oct 30 02:09:26.905064 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:26.904812 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-24fb97\"/\"default-dockercfg-5gtr5\"" Oct 30 02:09:26.950021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:26.949985 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:27.062999 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.062956 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbsw8\" (UniqueName: \"kubernetes.io/projected/0cff25a3-26e7-423f-a856-2568ebb922ca-kube-api-access-vbsw8\") pod \"auth-test-model-24fb97-kserve-d69bffd77-468np\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.063247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.063017 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-dshm\") pod \"auth-test-model-24fb97-kserve-d69bffd77-468np\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.063247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.063185 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-home\") pod \"auth-test-model-24fb97-kserve-d69bffd77-468np\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.063247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.063240 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cff25a3-26e7-423f-a856-2568ebb922ca-tls-certs\") pod \"auth-test-model-24fb97-kserve-d69bffd77-468np\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.063383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.063326 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-tmp-dir\") pod \"auth-test-model-24fb97-kserve-d69bffd77-468np\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.063383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.063373 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-model-cache\") pod \"auth-test-model-24fb97-kserve-d69bffd77-468np\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.172921 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.168191 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cff25a3-26e7-423f-a856-2568ebb922ca-tls-certs\") pod \"auth-test-model-24fb97-kserve-d69bffd77-468np\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.172921 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.168280 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-tmp-dir\") pod \"auth-test-model-24fb97-kserve-d69bffd77-468np\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.172921 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.168350 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-model-cache\") pod \"auth-test-model-24fb97-kserve-d69bffd77-468np\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.172921 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.168586 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbsw8\" (UniqueName: \"kubernetes.io/projected/0cff25a3-26e7-423f-a856-2568ebb922ca-kube-api-access-vbsw8\") pod \"auth-test-model-24fb97-kserve-d69bffd77-468np\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.172921 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.168624 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-dshm\") pod \"auth-test-model-24fb97-kserve-d69bffd77-468np\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.172921 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.168724 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-home\") pod \"auth-test-model-24fb97-kserve-d69bffd77-468np\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.172921 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.169351 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-home\") pod \"auth-test-model-24fb97-kserve-d69bffd77-468np\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.172921 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.169458 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-model-cache\") pod \"auth-test-model-24fb97-kserve-d69bffd77-468np\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.172921 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.170413 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-tmp-dir\") pod \"auth-test-model-24fb97-kserve-d69bffd77-468np\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.183697 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.183664 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cff25a3-26e7-423f-a856-2568ebb922ca-tls-certs\") pod \"auth-test-model-24fb97-kserve-d69bffd77-468np\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.213112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.209135 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-dshm\") pod \"auth-test-model-24fb97-kserve-d69bffd77-468np\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.227462 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.216388 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbsw8\" (UniqueName: \"kubernetes.io/projected/0cff25a3-26e7-423f-a856-2568ebb922ca-kube-api-access-vbsw8\") pod \"auth-test-model-24fb97-kserve-d69bffd77-468np\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.494954 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.494868 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:27.816493 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.816380 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np"] Oct 30 02:09:27.985527 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.985487 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" event={"ID":"0cff25a3-26e7-423f-a856-2568ebb922ca","Type":"ContainerStarted","Data":"764670faa98c76e9528660a8bbd4b2d874b0804de3f07fd927adaab1f344af1b"} Oct 30 02:09:27.985706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:27.985539 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" event={"ID":"0cff25a3-26e7-423f-a856-2568ebb922ca","Type":"ContainerStarted","Data":"e5d1a8d47dcb88ce7593ec1682a8fd42fc5ae4502028d331f2a2303026273a41"} Oct 30 02:09:28.025735 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:28.025543 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" podStartSLOduration=2.025522792 podStartE2EDuration="2.025522792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:09:28.02323925 +0000 UTC m=+2397.602030343" watchObservedRunningTime="2026-10-30 02:09:28.025522792 +0000 UTC m=+2397.604313876" Oct 30 02:09:28.188645 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:28.188610 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:28.495218 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:28.495144 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:29.873502 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:29.873466 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:30.002544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:30.002512 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:31.592358 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:31.592314 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:33.118687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:33.118646 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:33.155068 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:33.155029 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:09:34.041484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:34.041450 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp"] Oct 30 02:09:34.050759 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:34.050731 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83656716-h2nwp"] Oct 30 02:09:34.540429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:34.540391 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c8baaad-047f-4855-865c-ba855adb52c9" path="/var/lib/kubelet/pods/8c8baaad-047f-4855-865c-ba855adb52c9/volumes" Oct 30 02:09:34.957485 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:34.957445 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:36.240291 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:36.240259 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:36.353459 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:36.353415 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:36.749641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:36.749608 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:36.884649 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:36.884581 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:37.495929 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:37.495898 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:37.538776 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:37.538748 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:37.640908 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:37.640873 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:37.888474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:37.888380 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:38.185487 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:38.185397 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:38.290356 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:38.290324 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:09:38.892113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:38.889816 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:39.064120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:39.059916 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:39.248112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:39.247150 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:40.929064 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:40.928903 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:42.354862 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:42.354822 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:42.488326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:42.488260 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:42.833850 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:42.833804 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:44.050817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:44.050772 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:44.177729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:44.177687 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:44.861254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:44.861166 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:45.187706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:45.187674 5998 patch_prober.go:29] interesting pod/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 02:09:45.188184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:45.187743 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" podUID="29ec48a4-d344-4e9c-bba3-b1b68bc21a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 02:09:45.255887 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:45.255841 5998 generic.go:356] "Generic (PLEG): container finished" podID="29ec48a4-d344-4e9c-bba3-b1b68bc21a84" containerID="46d21d37f9becc1c0ef4e0a192c45cbe5dd8bfd2a9fb4829ac9d77ee63132ce5" exitCode=0 Oct 30 02:09:45.255887 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:45.255888 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" event={"ID":"29ec48a4-d344-4e9c-bba3-b1b68bc21a84","Type":"ContainerDied","Data":"46d21d37f9becc1c0ef4e0a192c45cbe5dd8bfd2a9fb4829ac9d77ee63132ce5"} Oct 30 02:09:45.256428 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:45.256411 5998 scope.go:122] "RemoveContainer" containerID="46d21d37f9becc1c0ef4e0a192c45cbe5dd8bfd2a9fb4829ac9d77ee63132ce5" Oct 30 02:09:45.258221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:45.258190 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:09:45.258300 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:45.258264 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:09:45.258300 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:45.258295 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:09:46.266291 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:46.266254 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" event={"ID":"29ec48a4-d344-4e9c-bba3-b1b68bc21a84","Type":"ContainerStarted","Data":"e9d2da0202a128ad214bf1e0c21da65eb30562d4b4d441932dd0f6bc7621096e"} Oct 30 02:09:46.756026 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:46.755983 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:46.837520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:46.837483 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:09:46.837889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:46.837866 5998 patch_prober.go:29] interesting pod/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 02:09:46.838042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:46.838017 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" podUID="29ec48a4-d344-4e9c-bba3-b1b68bc21a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 02:09:47.045373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:47.045280 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj"] Oct 30 02:09:47.054322 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:47.054280 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d397d80-sxgkj"] Oct 30 02:09:47.837512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:47.837478 5998 patch_prober.go:29] interesting pod/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 02:09:47.838007 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:47.837535 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" podUID="29ec48a4-d344-4e9c-bba3-b1b68bc21a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 02:09:48.412119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:48.406544 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:48.543843 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:48.543810 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd1b3bc6-a41a-4244-a7eb-3389898e88d2" path="/var/lib/kubelet/pods/cd1b3bc6-a41a-4244-a7eb-3389898e88d2/volumes" Oct 30 02:09:48.837596 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:48.837502 5998 patch_prober.go:29] interesting pod/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 02:09:48.838183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:48.838149 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" podUID="29ec48a4-d344-4e9c-bba3-b1b68bc21a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 02:09:49.558708 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.558673 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl"] Oct 30 02:09:49.564148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.562989 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.578385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.578356 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-594253\"/\"auth-test-model-594253-kserve-self-signed-certs\"" Oct 30 02:09:49.578776 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.578743 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-594253\"/\"openshift-service-ca.crt\"" Oct 30 02:09:49.594198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.594168 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl"] Oct 30 02:09:49.596992 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.596961 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxwvq\" (UniqueName: \"kubernetes.io/projected/696d7d69-7253-4856-b4bd-f0f650c0f2ce-kube-api-access-kxwvq\") pod \"auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.597151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.597021 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-dshm\") pod \"auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.597232 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.597199 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-home\") pod \"auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.597295 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.597228 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/696d7d69-7253-4856-b4bd-f0f650c0f2ce-tls-certs\") pod \"auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.597295 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.597263 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-tmp-dir\") pod \"auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.597402 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.597360 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-model-cache\") pod \"auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.647982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.647942 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:49.678681 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.678649 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-594253\"/\"default-dockercfg-hpjkc\"" Oct 30 02:09:49.680927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.680904 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-594253\"/\"kube-root-ca.crt\"" Oct 30 02:09:49.698651 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.698615 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxwvq\" (UniqueName: \"kubernetes.io/projected/696d7d69-7253-4856-b4bd-f0f650c0f2ce-kube-api-access-kxwvq\") pod \"auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.698798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.698671 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-dshm\") pod \"auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.698866 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.698800 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-home\") pod \"auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.698866 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.698833 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/696d7d69-7253-4856-b4bd-f0f650c0f2ce-tls-certs\") pod \"auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.698984 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.698879 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-tmp-dir\") pod \"auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.699039 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.699004 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-model-cache\") pod \"auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.699751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.699723 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-home\") pod \"auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.699892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.699766 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-tmp-dir\") pod \"auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.699892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.699815 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-model-cache\") pod \"auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.715486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.715440 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-dshm\") pod \"auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.727736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.727701 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/696d7d69-7253-4856-b4bd-f0f650c0f2ce-tls-certs\") pod \"auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.742877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.742844 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxwvq\" (UniqueName: \"kubernetes.io/projected/696d7d69-7253-4856-b4bd-f0f650c0f2ce-kube-api-access-kxwvq\") pod \"auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:49.844174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.843512 5998 patch_prober.go:29] interesting pod/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 02:09:49.844174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.843577 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" podUID="29ec48a4-d344-4e9c-bba3-b1b68bc21a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 02:09:49.887385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:49.887348 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:50.122364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:50.118823 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:50.372977 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:50.370275 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl"] Oct 30 02:09:50.375724 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:09:50.375688 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod696d7d69_7253_4856_b4bd_f0f650c0f2ce.slice/crio-9bfea2120d8b9b7437b3bb2c122b85f775355359eeb70de8393c0abf89699a90 WatchSource:0}: Error finding container 9bfea2120d8b9b7437b3bb2c122b85f775355359eeb70de8393c0abf89699a90: Status 404 returned error can't find the container with id 9bfea2120d8b9b7437b3bb2c122b85f775355359eeb70de8393c0abf89699a90 Oct 30 02:09:50.836797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:50.836758 5998 patch_prober.go:29] interesting pod/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 02:09:50.837002 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:50.836825 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" podUID="29ec48a4-d344-4e9c-bba3-b1b68bc21a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 02:09:51.041249 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:51.041209 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc"] Oct 30 02:09:51.045503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:51.045476 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e66dcbaa-7b4fc"] Oct 30 02:09:51.357885 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:51.357836 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" event={"ID":"696d7d69-7253-4856-b4bd-f0f650c0f2ce","Type":"ContainerStarted","Data":"e7d6d1a73d0835013ef3b126d50516286825b69eea221330d8506e314e008950"} Oct 30 02:09:51.358123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:51.357895 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" event={"ID":"696d7d69-7253-4856-b4bd-f0f650c0f2ce","Type":"ContainerStarted","Data":"9bfea2120d8b9b7437b3bb2c122b85f775355359eeb70de8393c0abf89699a90"} Oct 30 02:09:51.399040 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:51.398971 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" podStartSLOduration=2.398951881 podStartE2EDuration="2.398951881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:09:51.396850466 +0000 UTC m=+2420.975641549" watchObservedRunningTime="2026-10-30 02:09:51.398951881 +0000 UTC m=+2420.977742963" Oct 30 02:09:51.830189 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:51.824610 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:51.836653 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:51.836593 5998 patch_prober.go:29] interesting pod/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 02:09:51.836925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:51.836890 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" podUID="29ec48a4-d344-4e9c-bba3-b1b68bc21a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 02:09:51.887696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:51.887666 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:52.541248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:52.541212 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb00c574-0b03-48f4-a555-117ed93a9159" path="/var/lib/kubelet/pods/eb00c574-0b03-48f4-a555-117ed93a9159/volumes" Oct 30 02:09:52.841246 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:52.841155 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:09:52.842141 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:52.842110 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:09:52.842888 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:52.842867 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:09:53.773243 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:53.773206 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:55.222446 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:09:55.222371 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:09:55.366416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:55.366374 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:56.160613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:56.160565 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:57.441544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:57.441508 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:59.088527 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:59.088483 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:09:59.888520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:59.888482 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:09:59.901904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:09:59.901869 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:10:00.446709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:00.446676 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:10:00.720120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:00.719967 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:02.285001 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:02.284965 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:03.997080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:03.997043 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:04.045103 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:04.045036 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk"] Oct 30 02:10:04.065158 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:04.065121 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3955582b-tnglk"] Oct 30 02:10:04.539655 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:04.539620 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dded0acd-54b7-4148-a80a-f604e7722217" path="/var/lib/kubelet/pods/dded0acd-54b7-4148-a80a-f604e7722217/volumes" Oct 30 02:10:05.589377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:05.589346 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:07.464509 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:07.464472 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:07.578828 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:07.578785 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:07.703506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:07.703465 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:08.043941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:08.043901 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n"] Oct 30 02:10:08.053718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:08.053683 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2acf668b-sts4n"] Oct 30 02:10:08.543357 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:08.543316 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="225888f1-1466-4e02-a0e0-4a34e7ae1c28" path="/var/lib/kubelet/pods/225888f1-1466-4e02-a0e0-4a34e7ae1c28/volumes" Oct 30 02:10:09.482714 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:09.482662 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:11.116416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:11.116380 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:12.721902 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:12.721764 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:14.305634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:14.305590 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:15.972196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:15.972156 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:16.053951 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.053907 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj"] Oct 30 02:10:16.059281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.059256 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.080380 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.080350 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj"] Oct 30 02:10:16.086225 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.086198 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a1028d\"/\"default-dockercfg-5rz8m\"" Oct 30 02:10:16.086419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.086268 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a1028d\"/\"autores-model-a1028d-kserve-self-signed-certs\"" Oct 30 02:10:16.088034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.087700 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a1028d\"/\"openshift-service-ca.crt\"" Oct 30 02:10:16.088034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.087781 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a1028d\"/\"kube-root-ca.crt\"" Oct 30 02:10:16.125609 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.125567 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:16.175891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.175857 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-model-cache\") pod \"autores-model-a1028d-kserve-58675b588f-8rljj\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.176131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.175910 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-dshm\") pod \"autores-model-a1028d-kserve-58675b588f-8rljj\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.176226 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.176124 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxzzp\" (UniqueName: \"kubernetes.io/projected/9870c111-b5cb-4fb9-99d0-60ecf094a54d-kube-api-access-jxzzp\") pod \"autores-model-a1028d-kserve-58675b588f-8rljj\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.176226 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.176212 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9870c111-b5cb-4fb9-99d0-60ecf094a54d-tls-certs\") pod \"autores-model-a1028d-kserve-58675b588f-8rljj\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.176339 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.176313 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-tmp-dir\") pod \"autores-model-a1028d-kserve-58675b588f-8rljj\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.176460 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.176444 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-home\") pod \"autores-model-a1028d-kserve-58675b588f-8rljj\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.277845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.277755 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxzzp\" (UniqueName: \"kubernetes.io/projected/9870c111-b5cb-4fb9-99d0-60ecf094a54d-kube-api-access-jxzzp\") pod \"autores-model-a1028d-kserve-58675b588f-8rljj\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.277845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.277833 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9870c111-b5cb-4fb9-99d0-60ecf094a54d-tls-certs\") pod \"autores-model-a1028d-kserve-58675b588f-8rljj\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.278067 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.277890 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-tmp-dir\") pod \"autores-model-a1028d-kserve-58675b588f-8rljj\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.278067 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.277992 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-home\") pod \"autores-model-a1028d-kserve-58675b588f-8rljj\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.278067 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.278034 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-model-cache\") pod \"autores-model-a1028d-kserve-58675b588f-8rljj\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.278492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.278464 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-model-cache\") pod \"autores-model-a1028d-kserve-58675b588f-8rljj\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.278576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.278472 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-home\") pod \"autores-model-a1028d-kserve-58675b588f-8rljj\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.278576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.278472 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-tmp-dir\") pod \"autores-model-a1028d-kserve-58675b588f-8rljj\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.278654 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.278580 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-dshm\") pod \"autores-model-a1028d-kserve-58675b588f-8rljj\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.280698 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.280665 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9870c111-b5cb-4fb9-99d0-60ecf094a54d-tls-certs\") pod \"autores-model-a1028d-kserve-58675b588f-8rljj\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.280813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.280710 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-dshm\") pod \"autores-model-a1028d-kserve-58675b588f-8rljj\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.301444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.301396 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxzzp\" (UniqueName: \"kubernetes.io/projected/9870c111-b5cb-4fb9-99d0-60ecf094a54d-kube-api-access-jxzzp\") pod \"autores-model-a1028d-kserve-58675b588f-8rljj\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.375157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.374367 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:16.829412 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:16.829329 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj"] Oct 30 02:10:16.832784 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:10:16.832751 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9870c111_b5cb_4fb9_99d0_60ecf094a54d.slice/crio-eaa49248d1fb0903ae703761bff5d5ac3782159e6491ae7d02fbed1804cb2529 WatchSource:0}: Error finding container eaa49248d1fb0903ae703761bff5d5ac3782159e6491ae7d02fbed1804cb2529: Status 404 returned error can't find the container with id eaa49248d1fb0903ae703761bff5d5ac3782159e6491ae7d02fbed1804cb2529 Oct 30 02:10:17.042204 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:17.042167 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r"] Oct 30 02:10:17.049035 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:17.049005 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce2f3aab-cwt5r"] Oct 30 02:10:17.601885 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:17.601848 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" event={"ID":"9870c111-b5cb-4fb9-99d0-60ecf094a54d","Type":"ContainerStarted","Data":"b4f6a308cdfc1fb7a75fd73c39b757e5030434b00ad77cff072ea00665f92009"} Oct 30 02:10:17.602074 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:17.601901 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" event={"ID":"9870c111-b5cb-4fb9-99d0-60ecf094a54d","Type":"ContainerStarted","Data":"eaa49248d1fb0903ae703761bff5d5ac3782159e6491ae7d02fbed1804cb2529"} Oct 30 02:10:17.650247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:17.650184 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" podStartSLOduration=1.6501637200000001 podStartE2EDuration="1.65016372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:10:17.648032832 +0000 UTC m=+2447.226823913" watchObservedRunningTime="2026-10-30 02:10:17.65016372 +0000 UTC m=+2447.228954819" Oct 30 02:10:17.765892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:17.765862 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:18.036574 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:18.036540 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr"] Oct 30 02:10:18.042796 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:18.042764 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4de44e8-mjwlr"] Oct 30 02:10:18.375575 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:18.375497 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:18.544066 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:18.544033 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93fb3715-c2f0-48dc-b89b-a98d27b49440" path="/var/lib/kubelet/pods/93fb3715-c2f0-48dc-b89b-a98d27b49440/volumes" Oct 30 02:10:18.545204 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:18.544749 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eefe0f81-82c8-4491-90b9-09448b23b850" path="/var/lib/kubelet/pods/eefe0f81-82c8-4491-90b9-09448b23b850/volumes" Oct 30 02:10:19.401318 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:19.401266 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:21.259557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:21.259517 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:21.378557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:21.378499 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:22.871341 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:22.871304 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:23.862113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:23.861105 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:10:24.565136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:24.565073 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:25.225920 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:10:25.225882 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:10:26.375075 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:26.375039 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:26.389217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:26.389188 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:26.441641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:26.441605 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:26.709974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:26.709931 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:27.035387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:27.035300 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6"] Oct 30 02:10:27.039457 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:27.039425 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3054e243-d4rd6"] Oct 30 02:10:28.090065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:28.090031 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:28.544511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:28.544473 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead31f45-5cfc-4910-a2bd-1e4736f1268a" path="/var/lib/kubelet/pods/ead31f45-5cfc-4910-a2bd-1e4736f1268a/volumes" Oct 30 02:10:29.655465 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:29.655422 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:31.041944 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:31.041894 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298"] Oct 30 02:10:31.049724 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:31.049687 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b838ef26-v9298"] Oct 30 02:10:31.273054 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:31.272977 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:32.203830 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:32.203795 5998 scope.go:122] "RemoveContainer" containerID="b469979ed5190efdb5ad7f47aa71f54eeafaf4cd78d2513811348cfeff5aa0e6" Oct 30 02:10:32.542882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:32.542790 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e11889f6-1ef3-4f9e-80f0-131f9c19f2f6" path="/var/lib/kubelet/pods/e11889f6-1ef3-4f9e-80f0-131f9c19f2f6/volumes" Oct 30 02:10:32.895590 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:32.895497 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:34.576116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:34.564057 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:34.673672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:34.673637 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj"] Oct 30 02:10:34.855748 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:34.855663 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:35.082485 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:35.082446 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:35.649799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:35.649758 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:35.761626 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:35.761586 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:36.700535 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:36.700478 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:36.916833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:36.916795 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:38.372253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:38.372220 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:38.514591 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:38.514555 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:40.010078 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:40.010046 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:40.352708 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:40.352624 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:41.984285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:41.984216 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:43.036082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:43.036032 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t"] Oct 30 02:10:43.045954 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:43.045920 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ff95b39-rf42t"] Oct 30 02:10:43.574186 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:43.574150 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:44.539346 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:44.539302 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65158b49-402f-48d8-8335-e6cb600e299f" path="/var/lib/kubelet/pods/65158b49-402f-48d8-8335-e6cb600e299f/volumes" Oct 30 02:10:45.111321 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:45.111286 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:46.826028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:46.825992 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:48.056600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:48.056561 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k"] Oct 30 02:10:48.067682 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:48.067643 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81202a85-rjm6k"] Oct 30 02:10:48.509518 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:48.509477 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:48.543760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:48.543720 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bd1e229-67e3-4acd-8c95-e821a8812050" path="/var/lib/kubelet/pods/9bd1e229-67e3-4acd-8c95-e821a8812050/volumes" Oct 30 02:10:49.758153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:49.758080 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" podUID="9870c111-b5cb-4fb9-99d0-60ecf094a54d" containerName="main" containerID="cri-o://b4f6a308cdfc1fb7a75fd73c39b757e5030434b00ad77cff072ea00665f92009" gracePeriod=45 Oct 30 02:10:49.924779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:49.924739 5998 generic.go:356] "Generic (PLEG): container finished" podID="9870c111-b5cb-4fb9-99d0-60ecf094a54d" containerID="b4f6a308cdfc1fb7a75fd73c39b757e5030434b00ad77cff072ea00665f92009" exitCode=0 Oct 30 02:10:49.924951 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:49.924787 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" event={"ID":"9870c111-b5cb-4fb9-99d0-60ecf094a54d","Type":"ContainerDied","Data":"b4f6a308cdfc1fb7a75fd73c39b757e5030434b00ad77cff072ea00665f92009"} Oct 30 02:10:50.057705 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.057681 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:50.110712 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.110649 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:50.195422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.195387 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-tmp-dir\") pod \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " Oct 30 02:10:50.195635 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.195492 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-dshm\") pod \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " Oct 30 02:10:50.195635 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.195609 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9870c111-b5cb-4fb9-99d0-60ecf094a54d-kube-api-access-jxzzp\" (UniqueName: \"kubernetes.io/projected/9870c111-b5cb-4fb9-99d0-60ecf094a54d-kube-api-access-jxzzp\") pod \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " Oct 30 02:10:50.195762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.195697 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-home\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-home\") pod \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " Oct 30 02:10:50.195762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.195745 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-model-cache\") pod \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " Oct 30 02:10:50.195872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.195778 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-tmp-dir" pod "9870c111-b5cb-4fb9-99d0-60ecf094a54d" (UID: "9870c111-b5cb-4fb9-99d0-60ecf094a54d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:10:50.195872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.195813 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9870c111-b5cb-4fb9-99d0-60ecf094a54d-tls-certs\" (UniqueName: \"kubernetes.io/secret/9870c111-b5cb-4fb9-99d0-60ecf094a54d-tls-certs\") pod \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\" (UID: \"9870c111-b5cb-4fb9-99d0-60ecf094a54d\") " Oct 30 02:10:50.196029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.195997 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-model-cache" pod "9870c111-b5cb-4fb9-99d0-60ecf094a54d" (UID: "9870c111-b5cb-4fb9-99d0-60ecf094a54d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:10:50.197343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.196492 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:10:50.197343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.196518 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:10:50.197343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.196578 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-home" pod "9870c111-b5cb-4fb9-99d0-60ecf094a54d" (UID: "9870c111-b5cb-4fb9-99d0-60ecf094a54d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:10:50.199047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.198998 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9870c111-b5cb-4fb9-99d0-60ecf094a54d-tls-certs" pod "9870c111-b5cb-4fb9-99d0-60ecf094a54d" (UID: "9870c111-b5cb-4fb9-99d0-60ecf094a54d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:10:50.200070 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.200049 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-dshm" pod "9870c111-b5cb-4fb9-99d0-60ecf094a54d" (UID: "9870c111-b5cb-4fb9-99d0-60ecf094a54d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:10:50.202973 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.202937 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9870c111-b5cb-4fb9-99d0-60ecf094a54d-kube-api-access-jxzzp" pod "9870c111-b5cb-4fb9-99d0-60ecf094a54d" (UID: "9870c111-b5cb-4fb9-99d0-60ecf094a54d"). InnerVolumeSpecName "kube-api-access-jxzzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:10:50.297742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.297659 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:10:50.297742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.297689 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxzzp\" (UniqueName: \"kubernetes.io/projected/9870c111-b5cb-4fb9-99d0-60ecf094a54d-kube-api-access-jxzzp\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:10:50.297742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.297700 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9870c111-b5cb-4fb9-99d0-60ecf094a54d-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:10:50.297742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.297708 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9870c111-b5cb-4fb9-99d0-60ecf094a54d-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:10:50.934262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.934225 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" Oct 30 02:10:50.934262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.934237 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj" event={"ID":"9870c111-b5cb-4fb9-99d0-60ecf094a54d","Type":"ContainerDied","Data":"eaa49248d1fb0903ae703761bff5d5ac3782159e6491ae7d02fbed1804cb2529"} Oct 30 02:10:50.934831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.934296 5998 scope.go:122] "RemoveContainer" containerID="b4f6a308cdfc1fb7a75fd73c39b757e5030434b00ad77cff072ea00665f92009" Oct 30 02:10:50.987841 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.987811 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj"] Oct 30 02:10:50.996728 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:50.996698 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-a1028d/autores-model-a1028d-kserve-58675b588f-8rljj"] Oct 30 02:10:51.787936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:51.787907 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:52.541501 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:52.541457 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9870c111-b5cb-4fb9-99d0-60ecf094a54d" path="/var/lib/kubelet/pods/9870c111-b5cb-4fb9-99d0-60ecf094a54d/volumes" Oct 30 02:10:53.239136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:53.239103 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:10:53.523785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:53.523687 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:55.225110 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:10:55.225059 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:10:55.392437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:55.392400 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:57.005488 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:57.005454 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:58.092028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:58.091993 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9"] Oct 30 02:10:58.103777 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:58.103746 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-147ea4a5-wbtp9"] Oct 30 02:10:58.541160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:58.541125 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a844676-c921-41c0-a1f7-12d5c962b73e" path="/var/lib/kubelet/pods/1a844676-c921-41c0-a1f7-12d5c962b73e/volumes" Oct 30 02:10:58.985413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:58.985374 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq"] Oct 30 02:10:58.986323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:58.986298 5998 cpu_manager.go:414] "containerMap: removing container" podUID="9870c111-b5cb-4fb9-99d0-60ecf094a54d" containerName="main" Oct 30 02:10:58.986323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:58.986323 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9870c111-b5cb-4fb9-99d0-60ecf094a54d" containerName="main" Oct 30 02:10:58.986521 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:58.986508 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="9870c111-b5cb-4fb9-99d0-60ecf094a54d" containerName="main" Oct 30 02:10:58.990759 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:58.990725 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.020577 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.020542 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:59.027372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.027344 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-584ec5\"/\"default-dockercfg-drkbz\"" Oct 30 02:10:59.027571 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.027533 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-584ec5\"/\"openshift-service-ca.crt\"" Oct 30 02:10:59.027679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.027638 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-584ec5\"/\"auth-test-model-584ec5-kserve-self-signed-certs\"" Oct 30 02:10:59.027765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.027745 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-584ec5\"/\"kube-root-ca.crt\"" Oct 30 02:10:59.029079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.029047 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f35655-46ac-4a54-8326-077b1c659773-tls-certs\") pod \"auth-test-model-584ec5-kserve-8589c66d4f-psgsq\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.030702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.029183 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-home\") pod \"auth-test-model-584ec5-kserve-8589c66d4f-psgsq\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.030702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.029323 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-dshm\") pod \"auth-test-model-584ec5-kserve-8589c66d4f-psgsq\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.030702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.029367 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-model-cache\") pod \"auth-test-model-584ec5-kserve-8589c66d4f-psgsq\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.030702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.029408 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxhpn\" (UniqueName: \"kubernetes.io/projected/80f35655-46ac-4a54-8326-077b1c659773-kube-api-access-lxhpn\") pod \"auth-test-model-584ec5-kserve-8589c66d4f-psgsq\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.030702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.029451 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-tmp-dir\") pod \"auth-test-model-584ec5-kserve-8589c66d4f-psgsq\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.035527 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.035497 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq"] Oct 30 02:10:59.131700 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.131649 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-dshm\") pod \"auth-test-model-584ec5-kserve-8589c66d4f-psgsq\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.132237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.131718 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-model-cache\") pod \"auth-test-model-584ec5-kserve-8589c66d4f-psgsq\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.132237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.131757 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhpn\" (UniqueName: \"kubernetes.io/projected/80f35655-46ac-4a54-8326-077b1c659773-kube-api-access-lxhpn\") pod \"auth-test-model-584ec5-kserve-8589c66d4f-psgsq\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.132237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.131799 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-tmp-dir\") pod \"auth-test-model-584ec5-kserve-8589c66d4f-psgsq\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.132237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.131931 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f35655-46ac-4a54-8326-077b1c659773-tls-certs\") pod \"auth-test-model-584ec5-kserve-8589c66d4f-psgsq\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.132237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.132042 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-home\") pod \"auth-test-model-584ec5-kserve-8589c66d4f-psgsq\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.132547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.132307 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-tmp-dir\") pod \"auth-test-model-584ec5-kserve-8589c66d4f-psgsq\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.132547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.132471 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-home\") pod \"auth-test-model-584ec5-kserve-8589c66d4f-psgsq\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.132645 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.132544 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-model-cache\") pod \"auth-test-model-584ec5-kserve-8589c66d4f-psgsq\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.134618 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.134590 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-dshm\") pod \"auth-test-model-584ec5-kserve-8589c66d4f-psgsq\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.140400 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.140370 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f35655-46ac-4a54-8326-077b1c659773-tls-certs\") pod \"auth-test-model-584ec5-kserve-8589c66d4f-psgsq\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.160080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.160043 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxhpn\" (UniqueName: \"kubernetes.io/projected/80f35655-46ac-4a54-8326-077b1c659773-kube-api-access-lxhpn\") pod \"auth-test-model-584ec5-kserve-8589c66d4f-psgsq\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.214394 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.214351 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:10:59.324161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.312455 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:10:59.697119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:10:59.697065 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq"] Oct 30 02:10:59.699509 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:10:59.699484 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80f35655_46ac_4a54_8326_077b1c659773.slice/crio-29e0408950de6ed3faa436d29fb39f0564358c30b5a98fbc1c0b88392b288425 WatchSource:0}: Error finding container 29e0408950de6ed3faa436d29fb39f0564358c30b5a98fbc1c0b88392b288425: Status 404 returned error can't find the container with id 29e0408950de6ed3faa436d29fb39f0564358c30b5a98fbc1c0b88392b288425 Oct 30 02:11:00.013725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:00.013634 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" event={"ID":"80f35655-46ac-4a54-8326-077b1c659773","Type":"ContainerStarted","Data":"5e6474eeded05740c4a7d52e4f315322151df794102e2172ab21c524016dad65"} Oct 30 02:11:00.013725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:00.013689 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" event={"ID":"80f35655-46ac-4a54-8326-077b1c659773","Type":"ContainerStarted","Data":"29e0408950de6ed3faa436d29fb39f0564358c30b5a98fbc1c0b88392b288425"} Oct 30 02:11:00.056513 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:00.056455 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" podStartSLOduration=2.056435437 podStartE2EDuration="2.056435437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:11:00.052952438 +0000 UTC m=+2489.631743520" watchObservedRunningTime="2026-10-30 02:11:00.056435437 +0000 UTC m=+2489.635226519" Oct 30 02:11:00.313725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:00.313628 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:11:00.773300 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:00.773261 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:02.059780 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:02.059744 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:02.381495 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:02.381402 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:04.015942 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:04.015899 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:05.601525 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:05.601479 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:07.192960 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:07.192921 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:08.782825 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:08.782771 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:09.313162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:09.313118 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:11:09.325985 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:09.325959 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:11:10.128652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:10.128624 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:11:10.472685 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:10.472623 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:12.112383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:12.112337 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:14.124159 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:14.124080 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:15.037587 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:15.037548 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w"] Oct 30 02:11:15.047331 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:15.047299 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96fdbffa-g7v6w"] Oct 30 02:11:15.881543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:15.881509 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:16.556000 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:16.555956 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee1fda9-fa93-431b-a16b-8d0a2c520811" path="/var/lib/kubelet/pods/8ee1fda9-fa93-431b-a16b-8d0a2c520811/volumes" Oct 30 02:11:17.569589 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:17.569538 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:17.934453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:17.934412 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4"] Oct 30 02:11:17.944100 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:17.944057 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:17.956643 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:17.956613 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4"] Oct 30 02:11:17.959669 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:17.959645 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-01a9cd\"/\"kube-root-ca.crt\"" Oct 30 02:11:17.961479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:17.961451 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-01a9cd\"/\"openshift-service-ca.crt\"" Oct 30 02:11:17.961608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:17.961475 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-01a9cd\"/\"default-dockercfg-nscnx\"" Oct 30 02:11:17.970007 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:17.969978 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-01a9cd\"/\"auth-test-model-01a9cd-kserve-self-signed-certs\"" Oct 30 02:11:17.983303 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:17.983269 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:18.081580 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.081540 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-model-cache\") pod \"auth-test-model-01a9cd-kserve-d756499b5-rgxq4\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.081785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.081623 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ec2625a-1e27-4543-aa0b-352230fbe1a8-tls-certs\") pod \"auth-test-model-01a9cd-kserve-d756499b5-rgxq4\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.081785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.081678 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-tmp-dir\") pod \"auth-test-model-01a9cd-kserve-d756499b5-rgxq4\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.081920 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.081790 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-dshm\") pod \"auth-test-model-01a9cd-kserve-d756499b5-rgxq4\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.081920 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.081874 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pjdr\" (UniqueName: \"kubernetes.io/projected/5ec2625a-1e27-4543-aa0b-352230fbe1a8-kube-api-access-2pjdr\") pod \"auth-test-model-01a9cd-kserve-d756499b5-rgxq4\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.081920 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.081901 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-home\") pod \"auth-test-model-01a9cd-kserve-d756499b5-rgxq4\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.185194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.185069 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-model-cache\") pod \"auth-test-model-01a9cd-kserve-d756499b5-rgxq4\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.185517 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.185480 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ec2625a-1e27-4543-aa0b-352230fbe1a8-tls-certs\") pod \"auth-test-model-01a9cd-kserve-d756499b5-rgxq4\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.185659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.185553 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-tmp-dir\") pod \"auth-test-model-01a9cd-kserve-d756499b5-rgxq4\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.185725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.185700 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-dshm\") pod \"auth-test-model-01a9cd-kserve-d756499b5-rgxq4\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.185900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.185881 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pjdr\" (UniqueName: \"kubernetes.io/projected/5ec2625a-1e27-4543-aa0b-352230fbe1a8-kube-api-access-2pjdr\") pod \"auth-test-model-01a9cd-kserve-d756499b5-rgxq4\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.185998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.185964 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-home\") pod \"auth-test-model-01a9cd-kserve-d756499b5-rgxq4\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.188119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.186531 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-home\") pod \"auth-test-model-01a9cd-kserve-d756499b5-rgxq4\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.188119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.186824 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-model-cache\") pod \"auth-test-model-01a9cd-kserve-d756499b5-rgxq4\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.193053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.193028 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-tmp-dir\") pod \"auth-test-model-01a9cd-kserve-d756499b5-rgxq4\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.262116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.249062 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ec2625a-1e27-4543-aa0b-352230fbe1a8-tls-certs\") pod \"auth-test-model-01a9cd-kserve-d756499b5-rgxq4\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.262116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.253345 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-dshm\") pod \"auth-test-model-01a9cd-kserve-d756499b5-rgxq4\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.270041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.270003 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pjdr\" (UniqueName: \"kubernetes.io/projected/5ec2625a-1e27-4543-aa0b-352230fbe1a8-kube-api-access-2pjdr\") pod \"auth-test-model-01a9cd-kserve-d756499b5-rgxq4\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.562583 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.562497 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:18.876382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:18.876349 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4"] Oct 30 02:11:18.878879 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:11:18.878845 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ec2625a_1e27_4543_aa0b_352230fbe1a8.slice/crio-075e919427cedbc27b41d7d70c7f73ab0cc2a0638eb2837f7ea09e1cb755e2d3 WatchSource:0}: Error finding container 075e919427cedbc27b41d7d70c7f73ab0cc2a0638eb2837f7ea09e1cb755e2d3: Status 404 returned error can't find the container with id 075e919427cedbc27b41d7d70c7f73ab0cc2a0638eb2837f7ea09e1cb755e2d3 Oct 30 02:11:19.145060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:19.144932 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:19.260831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:19.260612 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:19.351691 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:19.351645 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" event={"ID":"5ec2625a-1e27-4543-aa0b-352230fbe1a8","Type":"ContainerStarted","Data":"fe6062dda7cfea711a7816ecbd6c44d8c5e90d8ea0b9ec252273aca4aaac0bc9"} Oct 30 02:11:19.351878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:19.351704 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" event={"ID":"5ec2625a-1e27-4543-aa0b-352230fbe1a8","Type":"ContainerStarted","Data":"075e919427cedbc27b41d7d70c7f73ab0cc2a0638eb2837f7ea09e1cb755e2d3"} Oct 30 02:11:19.398210 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:19.398114 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" podStartSLOduration=2.398081411 podStartE2EDuration="2.398081411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:11:19.394729118 +0000 UTC m=+2508.973520199" watchObservedRunningTime="2026-10-30 02:11:19.398081411 +0000 UTC m=+2508.976872489" Oct 30 02:11:19.475049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:19.475016 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:19.563333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:19.563302 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:19.791957 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:19.791919 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:21.152281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:21.152242 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:22.829029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:22.828933 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:23.039768 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:23.039731 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959"] Oct 30 02:11:23.049269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:23.049237 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32e3afef-q7959"] Oct 30 02:11:24.541897 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:24.541859 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2f3c8d5-6721-4780-a6dc-9d66c4785930" path="/var/lib/kubelet/pods/c2f3c8d5-6721-4780-a6dc-9d66c4785930/volumes" Oct 30 02:11:24.581415 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:24.581379 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:25.042208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:25.042167 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p"] Oct 30 02:11:25.048119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:25.048073 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-681f3da8-99c6p"] Oct 30 02:11:25.247900 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:11:25.247858 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:11:26.543661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:26.543619 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bdd3394-1315-4d86-988f-ab7fafafa831" path="/var/lib/kubelet/pods/7bdd3394-1315-4d86-988f-ab7fafafa831/volumes" Oct 30 02:11:26.637511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:26.637471 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:27.952494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:27.952458 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:28.233450 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:28.233359 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:28.563023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:28.562927 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:28.582268 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:28.582237 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:29.442507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:29.442463 5998 generic.go:356] "Generic (PLEG): container finished" podID="dcad96d8-5e9b-4083-bbb0-4533bfd7e813" containerID="5d13e8f6c074d33897484e0a2030b241276e138ff7c2519d20ae2332edab2cc9" exitCode=0 Oct 30 02:11:29.443338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:29.442542 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" event={"ID":"dcad96d8-5e9b-4083-bbb0-4533bfd7e813","Type":"ContainerDied","Data":"5d13e8f6c074d33897484e0a2030b241276e138ff7c2519d20ae2332edab2cc9"} Oct 30 02:11:29.443338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:29.443165 5998 scope.go:122] "RemoveContainer" containerID="5d13e8f6c074d33897484e0a2030b241276e138ff7c2519d20ae2332edab2cc9" Oct 30 02:11:29.444919 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:29.444889 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:11:29.445047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:29.444975 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:11:29.445047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:29.445024 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:11:29.456883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:29.456857 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:11:29.667588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:29.667559 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:11:29.789769 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:29.789678 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:30.034512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:30.034478 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44"] Oct 30 02:11:30.037845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:30.037817 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-324255df-cnd44"] Oct 30 02:11:30.456363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:30.456324 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" event={"ID":"dcad96d8-5e9b-4083-bbb0-4533bfd7e813","Type":"ContainerStarted","Data":"76786f424c282a68850ba9b005d169dc91097ea4adf469ca4c265c1f8dc54f73"} Oct 30 02:11:30.540780 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:30.540745 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="452f831c-f1d8-49df-900a-78f9a8972d98" path="/var/lib/kubelet/pods/452f831c-f1d8-49df-900a-78f9a8972d98/volumes" Oct 30 02:11:30.838613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:30.838523 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:11:30.839674 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:30.839645 5998 patch_prober.go:29] interesting pod/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 02:11:30.839825 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:30.839701 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" podUID="dcad96d8-5e9b-4083-bbb0-4533bfd7e813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 02:11:31.383894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:31.383847 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:31.838841 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:31.838793 5998 patch_prober.go:29] interesting pod/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 02:11:31.839312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:31.838857 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" podUID="dcad96d8-5e9b-4083-bbb0-4533bfd7e813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 02:11:32.247450 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:32.245552 5998 scope.go:122] "RemoveContainer" containerID="8a42562e87ae64fcced8b51cd9823f81bdc17241fa097e6e3638b02999a034d6" Oct 30 02:11:32.260778 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:32.260757 5998 scope.go:122] "RemoveContainer" containerID="d28cfdda14a0cf8bb4690873dbe60b4a4f2d397632a32f04449a893042f690d5" Oct 30 02:11:32.272300 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:32.272268 5998 scope.go:122] "RemoveContainer" containerID="1af10d8dfc3909b3eb2a99260f87be91a0be4d56b7284f91ba90ebdeccf2565f" Oct 30 02:11:32.289059 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:32.289033 5998 scope.go:122] "RemoveContainer" containerID="99ebdc01b16d1b695cfd53d7b5b0dbb2ad5cc93709762df201381c0701131e7c" Oct 30 02:11:32.298760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:32.298735 5998 scope.go:122] "RemoveContainer" containerID="e5813b1a1c0b84e143f61a4721a8ba8dfa08787fa2cd8649e22fcf2a3633dabc" Oct 30 02:11:32.339744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:32.339717 5998 scope.go:122] "RemoveContainer" containerID="89c2320aec011e56fa5b2a95da3713dded5c0fc3ba35b64e6bdebe070fcdc5e1" Oct 30 02:11:32.349445 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:32.349419 5998 scope.go:122] "RemoveContainer" containerID="de625751a900d92dc2b14e310a10d5d628f011dcf302176155d4a2b3b6333c18" Oct 30 02:11:32.359242 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:32.359222 5998 scope.go:122] "RemoveContainer" containerID="7812ef933f3b5ce6ed5828e49344d651c7cf9543dd9fd59228ac17bb385e41f1" Oct 30 02:11:32.836115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:32.835225 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:32.842115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:32.841884 5998 patch_prober.go:29] interesting pod/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 02:11:32.842115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:32.841948 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" podUID="dcad96d8-5e9b-4083-bbb0-4533bfd7e813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 02:11:33.626602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:33.626563 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:33.838687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:33.838650 5998 patch_prober.go:29] interesting pod/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 02:11:33.838863 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:33.838703 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" podUID="dcad96d8-5e9b-4083-bbb0-4533bfd7e813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 02:11:34.640320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:34.640286 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:34.747057 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:34.747021 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:34.838252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:34.838210 5998 patch_prober.go:29] interesting pod/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 30 02:11:34.838456 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:34.838274 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" podUID="dcad96d8-5e9b-4083-bbb0-4533bfd7e813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 30 02:11:35.396534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:35.396487 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:35.552571 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:35.552535 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:35.879251 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:35.873318 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5"] Oct 30 02:11:35.879251 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:35.873616 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" podUID="dcad96d8-5e9b-4083-bbb0-4533bfd7e813" containerName="istio-proxy" containerID="cri-o://76786f424c282a68850ba9b005d169dc91097ea4adf469ca4c265c1f8dc54f73" gracePeriod=30 Oct 30 02:11:36.012113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:36.010194 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:36.216907 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:36.216526 5998 patch_prober.go:29] interesting pod/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:11:36.216907 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:36.216598 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" podUID="dcad96d8-5e9b-4083-bbb0-4533bfd7e813" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:11:36.534393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:36.534298 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:37.087828 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:37.087786 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s"] Oct 30 02:11:37.099202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:37.099162 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5faf1a6-xmd9s"] Oct 30 02:11:37.154113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:37.144927 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:37.345168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:37.344970 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:37.485955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:37.485911 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:38.033901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:38.033867 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7"] Oct 30 02:11:38.045179 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:38.045145 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7957d8ce-qwqq7"] Oct 30 02:11:38.541050 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:38.541012 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba37437-81c8-4edf-97e2-f16aadb9eaa6" path="/var/lib/kubelet/pods/4ba37437-81c8-4edf-97e2-f16aadb9eaa6/volumes" Oct 30 02:11:38.541662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:38.541641 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9886070-5efc-43d6-a9bd-5a30014458ec" path="/var/lib/kubelet/pods/c9886070-5efc-43d6-a9bd-5a30014458ec/volumes" Oct 30 02:11:38.906550 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:38.906455 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:39.060105 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:39.060045 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:39.173103 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:39.173058 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:39.281620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:39.281591 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:40.868254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:40.868217 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:41.466299 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.466263 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:11:41.506657 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.506576 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-workload-socket\") pod \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " Oct 30 02:11:41.506818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.506748 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-podinfo\") pod \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " Oct 30 02:11:41.506896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.506818 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-token\" (UniqueName: \"kubernetes.io/projected/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-token\") pod \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " Oct 30 02:11:41.506896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.506863 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-workload-socket" pod "dcad96d8-5e9b-4083-bbb0-4533bfd7e813" (UID: "dcad96d8-5e9b-4083-bbb0-4533bfd7e813"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:41.506896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.506882 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-kube-api-access-rst7c\" (UniqueName: \"kubernetes.io/projected/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-kube-api-access-rst7c\") pod \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " Oct 30 02:11:41.507042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.506932 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-envoy\") pod \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " Oct 30 02:11:41.507042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.506964 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-data\") pod \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " Oct 30 02:11:41.507042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.507017 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-workload-certs\") pod \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " Oct 30 02:11:41.507238 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.507083 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istiod-ca-cert\") pod \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " Oct 30 02:11:41.508298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.507365 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-workload-certs" pod "dcad96d8-5e9b-4083-bbb0-4533bfd7e813" (UID: "dcad96d8-5e9b-4083-bbb0-4533bfd7e813"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:41.508298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.507428 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-credential-socket\") pod \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\" (UID: \"dcad96d8-5e9b-4083-bbb0-4533bfd7e813\") " Oct 30 02:11:41.508298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.507475 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-data" pod "dcad96d8-5e9b-4083-bbb0-4533bfd7e813" (UID: "dcad96d8-5e9b-4083-bbb0-4533bfd7e813"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:41.508298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.507758 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-credential-socket" pod "dcad96d8-5e9b-4083-bbb0-4533bfd7e813" (UID: "dcad96d8-5e9b-4083-bbb0-4533bfd7e813"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:41.508298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.507680 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istiod-ca-cert" pod "dcad96d8-5e9b-4083-bbb0-4533bfd7e813" (UID: "dcad96d8-5e9b-4083-bbb0-4533bfd7e813"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:11:41.508298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.508212 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:11:41.508298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.508235 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:11:41.508298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.508248 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:11:41.508298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.508261 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:11:41.508298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.508274 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:11:41.509494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.509457 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-token" pod "dcad96d8-5e9b-4083-bbb0-4533bfd7e813" (UID: "dcad96d8-5e9b-4083-bbb0-4533bfd7e813"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:41.509644 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.509622 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-kube-api-access-rst7c" pod "dcad96d8-5e9b-4083-bbb0-4533bfd7e813" (UID: "dcad96d8-5e9b-4083-bbb0-4533bfd7e813"). InnerVolumeSpecName "kube-api-access-rst7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:41.509966 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.509939 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-envoy" pod "dcad96d8-5e9b-4083-bbb0-4533bfd7e813" (UID: "dcad96d8-5e9b-4083-bbb0-4533bfd7e813"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:41.510477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.510450 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-podinfo" pod "dcad96d8-5e9b-4083-bbb0-4533bfd7e813" (UID: "dcad96d8-5e9b-4083-bbb0-4533bfd7e813"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:11:41.575737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.575698 5998 generic.go:356] "Generic (PLEG): container finished" podID="dcad96d8-5e9b-4083-bbb0-4533bfd7e813" containerID="76786f424c282a68850ba9b005d169dc91097ea4adf469ca4c265c1f8dc54f73" exitCode=0 Oct 30 02:11:41.576130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.575764 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" Oct 30 02:11:41.576130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.575789 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" event={"ID":"dcad96d8-5e9b-4083-bbb0-4533bfd7e813","Type":"ContainerDied","Data":"76786f424c282a68850ba9b005d169dc91097ea4adf469ca4c265c1f8dc54f73"} Oct 30 02:11:41.576130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.575849 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5" event={"ID":"dcad96d8-5e9b-4083-bbb0-4533bfd7e813","Type":"ContainerDied","Data":"8c597c68f4fde5f9dbcf4e20176a6dcfb7540c3f5095e51c9d7a5f0e8d6beeca"} Oct 30 02:11:41.576130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.575873 5998 scope.go:122] "RemoveContainer" containerID="76786f424c282a68850ba9b005d169dc91097ea4adf469ca4c265c1f8dc54f73" Oct 30 02:11:41.588494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.588475 5998 scope.go:122] "RemoveContainer" containerID="5d13e8f6c074d33897484e0a2030b241276e138ff7c2519d20ae2332edab2cc9" Oct 30 02:11:41.599054 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.599035 5998 scope.go:122] "RemoveContainer" containerID="76786f424c282a68850ba9b005d169dc91097ea4adf469ca4c265c1f8dc54f73" Oct 30 02:11:41.599384 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:11:41.599364 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76786f424c282a68850ba9b005d169dc91097ea4adf469ca4c265c1f8dc54f73\": container with ID starting with 76786f424c282a68850ba9b005d169dc91097ea4adf469ca4c265c1f8dc54f73 not found: ID does not exist" containerID="76786f424c282a68850ba9b005d169dc91097ea4adf469ca4c265c1f8dc54f73" Oct 30 02:11:41.599442 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.599397 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76786f424c282a68850ba9b005d169dc91097ea4adf469ca4c265c1f8dc54f73"} err="failed to get container status \"76786f424c282a68850ba9b005d169dc91097ea4adf469ca4c265c1f8dc54f73\": rpc error: code = NotFound desc = could not find container \"76786f424c282a68850ba9b005d169dc91097ea4adf469ca4c265c1f8dc54f73\": container with ID starting with 76786f424c282a68850ba9b005d169dc91097ea4adf469ca4c265c1f8dc54f73 not found: ID does not exist" Oct 30 02:11:41.599442 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.599413 5998 scope.go:122] "RemoveContainer" containerID="5d13e8f6c074d33897484e0a2030b241276e138ff7c2519d20ae2332edab2cc9" Oct 30 02:11:41.599673 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:11:41.599654 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d13e8f6c074d33897484e0a2030b241276e138ff7c2519d20ae2332edab2cc9\": container with ID starting with 5d13e8f6c074d33897484e0a2030b241276e138ff7c2519d20ae2332edab2cc9 not found: ID does not exist" containerID="5d13e8f6c074d33897484e0a2030b241276e138ff7c2519d20ae2332edab2cc9" Oct 30 02:11:41.599732 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.599683 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d13e8f6c074d33897484e0a2030b241276e138ff7c2519d20ae2332edab2cc9"} err="failed to get container status \"5d13e8f6c074d33897484e0a2030b241276e138ff7c2519d20ae2332edab2cc9\": rpc error: code = NotFound desc = could not find container \"5d13e8f6c074d33897484e0a2030b241276e138ff7c2519d20ae2332edab2cc9\": container with ID starting with 5d13e8f6c074d33897484e0a2030b241276e138ff7c2519d20ae2332edab2cc9 not found: ID does not exist" Oct 30 02:11:41.609667 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.609640 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:11:41.609667 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.609664 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:11:41.609872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.609674 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rst7c\" (UniqueName: \"kubernetes.io/projected/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-kube-api-access-rst7c\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:11:41.609872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.609685 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcad96d8-5e9b-4083-bbb0-4533bfd7e813-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:11:41.626540 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.626511 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5"] Oct 30 02:11:41.632025 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:41.631995 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-a1028d-openshift-default-98b848cc5-7gbg5"] Oct 30 02:11:42.372639 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:42.372598 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:42.544959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:42.544919 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcad96d8-5e9b-4083-bbb0-4533bfd7e813" path="/var/lib/kubelet/pods/dcad96d8-5e9b-4083-bbb0-4533bfd7e813/volumes" Oct 30 02:11:43.920128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:43.920060 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:45.582945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:45.582905 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:46.565811 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:46.565136 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:11:47.116114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:47.111222 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k"] Oct 30 02:11:47.140114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:47.133447 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" podUID="389bcbb5-189b-45d6-8154-e542936cf39b" containerName="istio-proxy" containerID="cri-o://c3edf767af4b3e3bd17eb839f79426cfc9ee1609559eda0bf9293052c0d3c21d" gracePeriod=30 Oct 30 02:11:47.434663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:47.434617 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:47.545838 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:47.545792 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:48.064574 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:48.064532 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr"] Oct 30 02:11:48.074329 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:48.074292 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e147c024-cwrpr"] Oct 30 02:11:48.539659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:48.539617 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96f5ee14-8359-4b68-83d5-8f71edca96ef" path="/var/lib/kubelet/pods/96f5ee14-8359-4b68-83d5-8f71edca96ef/volumes" Oct 30 02:11:49.173036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:49.172999 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:50.750882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:50.750843 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:51.045591 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:51.045469 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx"] Oct 30 02:11:51.053539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:51.053509 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e10aff-lfhcx"] Oct 30 02:11:52.436141 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.433425 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:52.533400 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.533326 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:11:52.543385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.543353 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee54a00-be9c-4ebd-bbd1-bdca4e49b70d" path="/var/lib/kubelet/pods/fee54a00-be9c-4ebd-bbd1-bdca4e49b70d/volumes" Oct 30 02:11:52.635065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.635018 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-istio-envoy\") pod \"389bcbb5-189b-45d6-8154-e542936cf39b\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " Oct 30 02:11:52.635308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.635072 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389bcbb5-189b-45d6-8154-e542936cf39b-kube-api-access-pbnqg\" (UniqueName: \"kubernetes.io/projected/389bcbb5-189b-45d6-8154-e542936cf39b-kube-api-access-pbnqg\") pod \"389bcbb5-189b-45d6-8154-e542936cf39b\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " Oct 30 02:11:52.635308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.635170 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-istio-data\") pod \"389bcbb5-189b-45d6-8154-e542936cf39b\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " Oct 30 02:11:52.635308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.635235 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-credential-socket\") pod \"389bcbb5-189b-45d6-8154-e542936cf39b\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " Oct 30 02:11:52.635498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.635313 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-workload-socket\") pod \"389bcbb5-189b-45d6-8154-e542936cf39b\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " Oct 30 02:11:52.635498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.635355 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-workload-certs\") pod \"389bcbb5-189b-45d6-8154-e542936cf39b\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " Oct 30 02:11:52.635498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.635384 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389bcbb5-189b-45d6-8154-e542936cf39b-istio-token\" (UniqueName: \"kubernetes.io/projected/389bcbb5-189b-45d6-8154-e542936cf39b-istio-token\") pod \"389bcbb5-189b-45d6-8154-e542936cf39b\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " Oct 30 02:11:52.635661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.635514 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/389bcbb5-189b-45d6-8154-e542936cf39b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389bcbb5-189b-45d6-8154-e542936cf39b-istiod-ca-cert\") pod \"389bcbb5-189b-45d6-8154-e542936cf39b\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " Oct 30 02:11:52.635661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.635580 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/389bcbb5-189b-45d6-8154-e542936cf39b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389bcbb5-189b-45d6-8154-e542936cf39b-istio-podinfo\") pod \"389bcbb5-189b-45d6-8154-e542936cf39b\" (UID: \"389bcbb5-189b-45d6-8154-e542936cf39b\") " Oct 30 02:11:52.636403 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.636072 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-credential-socket" pod "389bcbb5-189b-45d6-8154-e542936cf39b" (UID: "389bcbb5-189b-45d6-8154-e542936cf39b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:52.636403 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.636319 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:11:52.636586 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.636455 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-istio-data" pod "389bcbb5-189b-45d6-8154-e542936cf39b" (UID: "389bcbb5-189b-45d6-8154-e542936cf39b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:52.637146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.637011 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389bcbb5-189b-45d6-8154-e542936cf39b-istiod-ca-cert" pod "389bcbb5-189b-45d6-8154-e542936cf39b" (UID: "389bcbb5-189b-45d6-8154-e542936cf39b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:11:52.637146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.637111 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-workload-socket" pod "389bcbb5-189b-45d6-8154-e542936cf39b" (UID: "389bcbb5-189b-45d6-8154-e542936cf39b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:52.637695 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.637656 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-workload-certs" pod "389bcbb5-189b-45d6-8154-e542936cf39b" (UID: "389bcbb5-189b-45d6-8154-e542936cf39b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:52.640470 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.640435 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-istio-envoy" pod "389bcbb5-189b-45d6-8154-e542936cf39b" (UID: "389bcbb5-189b-45d6-8154-e542936cf39b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:11:52.642567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.642540 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389bcbb5-189b-45d6-8154-e542936cf39b-kube-api-access-pbnqg" pod "389bcbb5-189b-45d6-8154-e542936cf39b" (UID: "389bcbb5-189b-45d6-8154-e542936cf39b"). InnerVolumeSpecName "kube-api-access-pbnqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:52.642673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.642653 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/389bcbb5-189b-45d6-8154-e542936cf39b-istio-podinfo" pod "389bcbb5-189b-45d6-8154-e542936cf39b" (UID: "389bcbb5-189b-45d6-8154-e542936cf39b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:11:52.646994 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.644741 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389bcbb5-189b-45d6-8154-e542936cf39b-istio-token" pod "389bcbb5-189b-45d6-8154-e542936cf39b" (UID: "389bcbb5-189b-45d6-8154-e542936cf39b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:52.686333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.686288 5998 generic.go:356] "Generic (PLEG): container finished" podID="389bcbb5-189b-45d6-8154-e542936cf39b" containerID="c3edf767af4b3e3bd17eb839f79426cfc9ee1609559eda0bf9293052c0d3c21d" exitCode=0 Oct 30 02:11:52.686333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.686331 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" event={"ID":"389bcbb5-189b-45d6-8154-e542936cf39b","Type":"ContainerDied","Data":"c3edf767af4b3e3bd17eb839f79426cfc9ee1609559eda0bf9293052c0d3c21d"} Oct 30 02:11:52.686553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.686357 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" Oct 30 02:11:52.686553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.686372 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k" event={"ID":"389bcbb5-189b-45d6-8154-e542936cf39b","Type":"ContainerDied","Data":"cfbc5e62f2d555a04072f42722d5088877da0b7485853d6e655408f14da984c0"} Oct 30 02:11:52.686553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.686395 5998 scope.go:122] "RemoveContainer" containerID="c3edf767af4b3e3bd17eb839f79426cfc9ee1609559eda0bf9293052c0d3c21d" Oct 30 02:11:52.708301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.708273 5998 scope.go:122] "RemoveContainer" containerID="c3edf767af4b3e3bd17eb839f79426cfc9ee1609559eda0bf9293052c0d3c21d" Oct 30 02:11:52.708729 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:11:52.708695 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3edf767af4b3e3bd17eb839f79426cfc9ee1609559eda0bf9293052c0d3c21d\": container with ID starting with c3edf767af4b3e3bd17eb839f79426cfc9ee1609559eda0bf9293052c0d3c21d not found: ID does not exist" containerID="c3edf767af4b3e3bd17eb839f79426cfc9ee1609559eda0bf9293052c0d3c21d" Oct 30 02:11:52.708822 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.708742 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3edf767af4b3e3bd17eb839f79426cfc9ee1609559eda0bf9293052c0d3c21d"} err="failed to get container status \"c3edf767af4b3e3bd17eb839f79426cfc9ee1609559eda0bf9293052c0d3c21d\": rpc error: code = NotFound desc = could not find container \"c3edf767af4b3e3bd17eb839f79426cfc9ee1609559eda0bf9293052c0d3c21d\": container with ID starting with c3edf767af4b3e3bd17eb839f79426cfc9ee1609559eda0bf9293052c0d3c21d not found: ID does not exist" Oct 30 02:11:52.737247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.737215 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389bcbb5-189b-45d6-8154-e542936cf39b-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:11:52.737247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.737246 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:11:52.737464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.737259 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbnqg\" (UniqueName: \"kubernetes.io/projected/389bcbb5-189b-45d6-8154-e542936cf39b-kube-api-access-pbnqg\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:11:52.737464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.737272 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:11:52.737464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.737284 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:11:52.737464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.737294 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389bcbb5-189b-45d6-8154-e542936cf39b-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:11:52.737464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.737306 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389bcbb5-189b-45d6-8154-e542936cf39b-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:11:52.737464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.737319 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389bcbb5-189b-45d6-8154-e542936cf39b-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:11:52.748677 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.748645 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k"] Oct 30 02:11:52.755066 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:52.755036 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-dh29k"] Oct 30 02:11:54.062725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:54.062689 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:54.541566 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:54.541527 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="389bcbb5-189b-45d6-8154-e542936cf39b" path="/var/lib/kubelet/pods/389bcbb5-189b-45d6-8154-e542936cf39b/volumes" Oct 30 02:11:55.229837 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:11:55.229797 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:11:55.585594 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:55.585505 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:57.297098 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:57.297049 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:58.962065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:58.962028 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:11:59.037488 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:59.037448 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j"] Oct 30 02:11:59.046291 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:11:59.046256 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac627e96-hbd7j"] Oct 30 02:12:00.541171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:00.541137 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e326b0d1-a861-4e96-b418-b9ad631f6a33" path="/var/lib/kubelet/pods/e326b0d1-a861-4e96-b418-b9ad631f6a33/volumes" Oct 30 02:12:00.572436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:00.572396 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:02.195226 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:02.195189 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:03.770040 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:03.770000 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:05.324783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:05.324745 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:06.449584 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:06.449550 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:06.617936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:06.615345 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:06.941304 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:06.941263 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:07.173915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:07.173866 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:07.280811 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:07.280734 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:08.601523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:08.601484 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:10.183295 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:10.183225 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:11.871436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:11.871390 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:12.144706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:12.144616 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:13.481478 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:13.481445 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:14.045710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:14.045672 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4"] Oct 30 02:12:14.055501 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:14.055471 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db6a5ee-cgwj4"] Oct 30 02:12:14.541504 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:14.541459 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88548125-0316-444c-bf04-fbc0fbe9bbd4" path="/var/lib/kubelet/pods/88548125-0316-444c-bf04-fbc0fbe9bbd4/volumes" Oct 30 02:12:15.120148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:15.120083 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:16.677152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:16.677105 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:18.293032 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:18.292996 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:19.983518 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:19.983482 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:20.051464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:20.051422 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:12:21.740995 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:21.740955 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:23.026997 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.026960 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq"] Oct 30 02:12:23.027625 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.027606 5998 cpu_manager.go:414] "containerMap: removing container" podUID="dcad96d8-5e9b-4083-bbb0-4533bfd7e813" containerName="istio-proxy" Oct 30 02:12:23.027677 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.027626 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcad96d8-5e9b-4083-bbb0-4533bfd7e813" containerName="istio-proxy" Oct 30 02:12:23.027677 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.027648 5998 cpu_manager.go:414] "containerMap: removing container" podUID="389bcbb5-189b-45d6-8154-e542936cf39b" containerName="istio-proxy" Oct 30 02:12:23.027677 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.027653 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389bcbb5-189b-45d6-8154-e542936cf39b" containerName="istio-proxy" Oct 30 02:12:23.027677 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.027665 5998 cpu_manager.go:414] "containerMap: removing container" podUID="dcad96d8-5e9b-4083-bbb0-4533bfd7e813" containerName="istio-proxy" Oct 30 02:12:23.027677 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.027670 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcad96d8-5e9b-4083-bbb0-4533bfd7e813" containerName="istio-proxy" Oct 30 02:12:23.027835 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.027771 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcad96d8-5e9b-4083-bbb0-4533bfd7e813" containerName="istio-proxy" Oct 30 02:12:23.027835 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.027780 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="389bcbb5-189b-45d6-8154-e542936cf39b" containerName="istio-proxy" Oct 30 02:12:23.027835 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.027787 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcad96d8-5e9b-4083-bbb0-4533bfd7e813" containerName="istio-proxy" Oct 30 02:12:23.032548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.032527 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq" Oct 30 02:12:23.048909 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.048875 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:12:23.058180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.058157 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:12:23.058345 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.058170 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:12:23.058345 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.058244 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:12:23.063486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.063410 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq"] Oct 30 02:12:23.201072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.201037 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40108afb-04c9-46e4-9f0f-ef20676eb718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4d1932a-dsnwq\" (UID: \"40108afb-04c9-46e4-9f0f-ef20676eb718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq" Oct 30 02:12:23.201262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.201083 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqf5n\" (UniqueName: \"kubernetes.io/projected/40108afb-04c9-46e4-9f0f-ef20676eb718-kube-api-access-zqf5n\") pod \"maas-api-revoke-sub-d4d1932a-dsnwq\" (UID: \"40108afb-04c9-46e4-9f0f-ef20676eb718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq" Oct 30 02:12:23.201335 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.201315 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40108afb-04c9-46e4-9f0f-ef20676eb718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4d1932a-dsnwq\" (UID: \"40108afb-04c9-46e4-9f0f-ef20676eb718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq" Oct 30 02:12:23.302642 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.302562 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40108afb-04c9-46e4-9f0f-ef20676eb718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4d1932a-dsnwq\" (UID: \"40108afb-04c9-46e4-9f0f-ef20676eb718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq" Oct 30 02:12:23.302810 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.302708 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40108afb-04c9-46e4-9f0f-ef20676eb718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4d1932a-dsnwq\" (UID: \"40108afb-04c9-46e4-9f0f-ef20676eb718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq" Oct 30 02:12:23.302810 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.302758 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqf5n\" (UniqueName: \"kubernetes.io/projected/40108afb-04c9-46e4-9f0f-ef20676eb718-kube-api-access-zqf5n\") pod \"maas-api-revoke-sub-d4d1932a-dsnwq\" (UID: \"40108afb-04c9-46e4-9f0f-ef20676eb718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq" Oct 30 02:12:23.303415 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.303386 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40108afb-04c9-46e4-9f0f-ef20676eb718-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4d1932a-dsnwq\" (UID: \"40108afb-04c9-46e4-9f0f-ef20676eb718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq" Oct 30 02:12:23.305386 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.305359 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40108afb-04c9-46e4-9f0f-ef20676eb718-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4d1932a-dsnwq\" (UID: \"40108afb-04c9-46e4-9f0f-ef20676eb718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq" Oct 30 02:12:23.324316 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.324289 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqf5n\" (UniqueName: \"kubernetes.io/projected/40108afb-04c9-46e4-9f0f-ef20676eb718-kube-api-access-zqf5n\") pod \"maas-api-revoke-sub-d4d1932a-dsnwq\" (UID: \"40108afb-04c9-46e4-9f0f-ef20676eb718\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq" Oct 30 02:12:23.350377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.350339 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq" Oct 30 02:12:23.364266 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.364236 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:23.682153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.682120 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq"] Oct 30 02:12:23.684583 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:12:23.684552 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40108afb_04c9_46e4_9f0f_ef20676eb718.slice/crio-cd13465f49bd1bf0a045d62b43227ab086ded34857ea89776b244ffb40c75737 WatchSource:0}: Error finding container cd13465f49bd1bf0a045d62b43227ab086ded34857ea89776b244ffb40c75737: Status 404 returned error can't find the container with id cd13465f49bd1bf0a045d62b43227ab086ded34857ea89776b244ffb40c75737 Oct 30 02:12:23.984235 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:23.984134 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq" event={"ID":"40108afb-04c9-46e4-9f0f-ef20676eb718","Type":"ContainerStarted","Data":"cd13465f49bd1bf0a045d62b43227ab086ded34857ea89776b244ffb40c75737"} Oct 30 02:12:24.928969 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:24.928931 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:25.011784 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:25.011741 5998 generic.go:356] "Generic (PLEG): container finished" podID="40108afb-04c9-46e4-9f0f-ef20676eb718" containerID="f891d4a7ef591e5c691a64019ef5de17613ae4529a3d6440c9ea589d1531e33a" exitCode=0 Oct 30 02:12:25.011998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:25.011871 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq" event={"ID":"40108afb-04c9-46e4-9f0f-ef20676eb718","Type":"ContainerDied","Data":"f891d4a7ef591e5c691a64019ef5de17613ae4529a3d6440c9ea589d1531e33a"} Oct 30 02:12:25.228348 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:12:25.228250 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:12:26.195996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:26.195966 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq" Oct 30 02:12:26.258172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:26.258141 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40108afb-04c9-46e4-9f0f-ef20676eb718-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40108afb-04c9-46e4-9f0f-ef20676eb718-maas-api-openshift-ca\") pod \"40108afb-04c9-46e4-9f0f-ef20676eb718\" (UID: \"40108afb-04c9-46e4-9f0f-ef20676eb718\") " Oct 30 02:12:26.258384 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:26.258221 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40108afb-04c9-46e4-9f0f-ef20676eb718-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40108afb-04c9-46e4-9f0f-ef20676eb718-maas-api-xks-ca\") pod \"40108afb-04c9-46e4-9f0f-ef20676eb718\" (UID: \"40108afb-04c9-46e4-9f0f-ef20676eb718\") " Oct 30 02:12:26.258384 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:26.258328 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40108afb-04c9-46e4-9f0f-ef20676eb718-kube-api-access-zqf5n\" (UniqueName: \"kubernetes.io/projected/40108afb-04c9-46e4-9f0f-ef20676eb718-kube-api-access-zqf5n\") pod \"40108afb-04c9-46e4-9f0f-ef20676eb718\" (UID: \"40108afb-04c9-46e4-9f0f-ef20676eb718\") " Oct 30 02:12:26.258619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:26.258592 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40108afb-04c9-46e4-9f0f-ef20676eb718-maas-api-openshift-ca" pod "40108afb-04c9-46e4-9f0f-ef20676eb718" (UID: "40108afb-04c9-46e4-9f0f-ef20676eb718"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:12:26.259177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:26.259081 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40108afb-04c9-46e4-9f0f-ef20676eb718-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:26.261119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:26.260745 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40108afb-04c9-46e4-9f0f-ef20676eb718-maas-api-xks-ca" pod "40108afb-04c9-46e4-9f0f-ef20676eb718" (UID: "40108afb-04c9-46e4-9f0f-ef20676eb718"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:12:26.261119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:26.260872 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40108afb-04c9-46e4-9f0f-ef20676eb718-kube-api-access-zqf5n" pod "40108afb-04c9-46e4-9f0f-ef20676eb718" (UID: "40108afb-04c9-46e4-9f0f-ef20676eb718"). InnerVolumeSpecName "kube-api-access-zqf5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:26.359945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:26.359902 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40108afb-04c9-46e4-9f0f-ef20676eb718-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:26.359945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:26.359936 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqf5n\" (UniqueName: \"kubernetes.io/projected/40108afb-04c9-46e4-9f0f-ef20676eb718-kube-api-access-zqf5n\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:26.559082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:26.559052 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:27.028691 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:27.028660 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq" Oct 30 02:12:27.028902 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:27.028696 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq" event={"ID":"40108afb-04c9-46e4-9f0f-ef20676eb718","Type":"ContainerDied","Data":"cd13465f49bd1bf0a045d62b43227ab086ded34857ea89776b244ffb40c75737"} Oct 30 02:12:27.028902 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:27.028757 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd13465f49bd1bf0a045d62b43227ab086ded34857ea89776b244ffb40c75737" Oct 30 02:12:28.030679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:28.030636 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:29.528372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:29.528334 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:31.090719 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:31.090685 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:32.456429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:32.456394 5998 scope.go:122] "RemoveContainer" containerID="d6d654aa136d66e7c35fdab92405ccac9c57e9ff7d75d350ee41e4b9ca93cfcd" Oct 30 02:12:32.466892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:32.466868 5998 scope.go:122] "RemoveContainer" containerID="1f4b6bc6a74a9172ec1c03eec1b251995884579951c6baa7bce2366ce5aa17b8" Oct 30 02:12:32.476556 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:32.476537 5998 scope.go:122] "RemoveContainer" containerID="9296e05d2e83f2170ea7fb49622fcb857d932200ec65d7477051b3fca4086f10" Oct 30 02:12:32.485490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:32.485473 5998 scope.go:122] "RemoveContainer" containerID="1b0aa83df763c0c5b32fe3271a3dda6bad95836ef4f38b8c7b7d525bf3d9ca53" Oct 30 02:12:32.494895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:32.494872 5998 scope.go:122] "RemoveContainer" containerID="d51fc1f8096f35f11ea343e4ae01260202c2879fb8f054a73ef0984db8268e6d" Oct 30 02:12:32.504396 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:32.504378 5998 scope.go:122] "RemoveContainer" containerID="8c6c248a5bddfe7ee9f7c77488c238bab6957e1c7d3d0588b2965e303e6d52c9" Oct 30 02:12:32.513762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:32.513728 5998 scope.go:122] "RemoveContainer" containerID="8d7af1a7b2b710597fce498045890efa229ee54491a43d602d9d68ee9d61dc78" Oct 30 02:12:32.683148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:32.683106 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:33.346014 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.345973 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv"] Oct 30 02:12:33.346834 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.346814 5998 cpu_manager.go:414] "containerMap: removing container" podUID="40108afb-04c9-46e4-9f0f-ef20676eb718" containerName="revoke-keys" Oct 30 02:12:33.346906 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.346837 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40108afb-04c9-46e4-9f0f-ef20676eb718" containerName="revoke-keys" Oct 30 02:12:33.347050 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.347038 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="40108afb-04c9-46e4-9f0f-ef20676eb718" containerName="revoke-keys" Oct 30 02:12:33.353185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.353156 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv" Oct 30 02:12:33.366177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.366152 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:12:33.366303 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.366263 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:12:33.368357 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.368334 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv"] Oct 30 02:12:33.378131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.375178 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:12:33.378131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.375240 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:12:33.466019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.465984 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101c9f6c-b9b6-4408-873d-b0d14d3529b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd779be9-5fggv\" (UID: \"101c9f6c-b9b6-4408-873d-b0d14d3529b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv" Oct 30 02:12:33.466523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.466049 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101c9f6c-b9b6-4408-873d-b0d14d3529b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd779be9-5fggv\" (UID: \"101c9f6c-b9b6-4408-873d-b0d14d3529b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv" Oct 30 02:12:33.466523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.466296 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8ktz\" (UniqueName: \"kubernetes.io/projected/101c9f6c-b9b6-4408-873d-b0d14d3529b5-kube-api-access-d8ktz\") pod \"maas-api-revoke-sub-cd779be9-5fggv\" (UID: \"101c9f6c-b9b6-4408-873d-b0d14d3529b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv" Oct 30 02:12:33.567991 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.567944 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101c9f6c-b9b6-4408-873d-b0d14d3529b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd779be9-5fggv\" (UID: \"101c9f6c-b9b6-4408-873d-b0d14d3529b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv" Oct 30 02:12:33.568215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.568033 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101c9f6c-b9b6-4408-873d-b0d14d3529b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd779be9-5fggv\" (UID: \"101c9f6c-b9b6-4408-873d-b0d14d3529b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv" Oct 30 02:12:33.568215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.568178 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8ktz\" (UniqueName: \"kubernetes.io/projected/101c9f6c-b9b6-4408-873d-b0d14d3529b5-kube-api-access-d8ktz\") pod \"maas-api-revoke-sub-cd779be9-5fggv\" (UID: \"101c9f6c-b9b6-4408-873d-b0d14d3529b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv" Oct 30 02:12:33.568733 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.568703 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101c9f6c-b9b6-4408-873d-b0d14d3529b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd779be9-5fggv\" (UID: \"101c9f6c-b9b6-4408-873d-b0d14d3529b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv" Oct 30 02:12:33.570629 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.570603 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101c9f6c-b9b6-4408-873d-b0d14d3529b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd779be9-5fggv\" (UID: \"101c9f6c-b9b6-4408-873d-b0d14d3529b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv" Oct 30 02:12:33.590345 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.590321 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8ktz\" (UniqueName: \"kubernetes.io/projected/101c9f6c-b9b6-4408-873d-b0d14d3529b5-kube-api-access-d8ktz\") pod \"maas-api-revoke-sub-cd779be9-5fggv\" (UID: \"101c9f6c-b9b6-4408-873d-b0d14d3529b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv" Oct 30 02:12:33.670264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.670232 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv" Oct 30 02:12:33.980013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:33.979986 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv"] Oct 30 02:12:33.982695 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:12:33.982652 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101c9f6c_b9b6_4408_873d_b0d14d3529b5.slice/crio-8ce25c572db4e0fdd98309fcc5e3c3fd2956d219b4c11309412fbe0e82835bca WatchSource:0}: Error finding container 8ce25c572db4e0fdd98309fcc5e3c3fd2956d219b4c11309412fbe0e82835bca: Status 404 returned error can't find the container with id 8ce25c572db4e0fdd98309fcc5e3c3fd2956d219b4c11309412fbe0e82835bca Oct 30 02:12:34.032586 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:34.032559 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv"] Oct 30 02:12:34.036154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:34.036125 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20e36cb2-6wmpv"] Oct 30 02:12:34.090497 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:34.090462 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv" event={"ID":"101c9f6c-b9b6-4408-873d-b0d14d3529b5","Type":"ContainerStarted","Data":"8ce25c572db4e0fdd98309fcc5e3c3fd2956d219b4c11309412fbe0e82835bca"} Oct 30 02:12:34.185605 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:34.185553 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:34.541446 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:34.541414 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a541f1e-8c33-4f14-95f1-98ec0f568665" path="/var/lib/kubelet/pods/1a541f1e-8c33-4f14-95f1-98ec0f568665/volumes" Oct 30 02:12:35.100506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:35.100473 5998 generic.go:356] "Generic (PLEG): container finished" podID="101c9f6c-b9b6-4408-873d-b0d14d3529b5" containerID="f45ac764aab3c576a8905862416bf6206ecc7247be3ad1c9970d967659fd6bd8" exitCode=0 Oct 30 02:12:35.100699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:35.100525 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv" event={"ID":"101c9f6c-b9b6-4408-873d-b0d14d3529b5","Type":"ContainerDied","Data":"f45ac764aab3c576a8905862416bf6206ecc7247be3ad1c9970d967659fd6bd8"} Oct 30 02:12:35.766596 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:35.766548 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:36.290847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:36.290823 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv" Oct 30 02:12:36.413393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:36.413355 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101c9f6c-b9b6-4408-873d-b0d14d3529b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101c9f6c-b9b6-4408-873d-b0d14d3529b5-maas-api-openshift-ca\") pod \"101c9f6c-b9b6-4408-873d-b0d14d3529b5\" (UID: \"101c9f6c-b9b6-4408-873d-b0d14d3529b5\") " Oct 30 02:12:36.413579 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:36.413444 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101c9f6c-b9b6-4408-873d-b0d14d3529b5-kube-api-access-d8ktz\" (UniqueName: \"kubernetes.io/projected/101c9f6c-b9b6-4408-873d-b0d14d3529b5-kube-api-access-d8ktz\") pod \"101c9f6c-b9b6-4408-873d-b0d14d3529b5\" (UID: \"101c9f6c-b9b6-4408-873d-b0d14d3529b5\") " Oct 30 02:12:36.413627 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:36.413586 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/101c9f6c-b9b6-4408-873d-b0d14d3529b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101c9f6c-b9b6-4408-873d-b0d14d3529b5-maas-api-xks-ca\") pod \"101c9f6c-b9b6-4408-873d-b0d14d3529b5\" (UID: \"101c9f6c-b9b6-4408-873d-b0d14d3529b5\") " Oct 30 02:12:36.413822 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:36.413792 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101c9f6c-b9b6-4408-873d-b0d14d3529b5-maas-api-openshift-ca" pod "101c9f6c-b9b6-4408-873d-b0d14d3529b5" (UID: "101c9f6c-b9b6-4408-873d-b0d14d3529b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:12:36.414283 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:36.414262 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101c9f6c-b9b6-4408-873d-b0d14d3529b5-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:36.415666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:36.415645 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101c9f6c-b9b6-4408-873d-b0d14d3529b5-kube-api-access-d8ktz" pod "101c9f6c-b9b6-4408-873d-b0d14d3529b5" (UID: "101c9f6c-b9b6-4408-873d-b0d14d3529b5"). InnerVolumeSpecName "kube-api-access-d8ktz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:36.415803 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:36.415787 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101c9f6c-b9b6-4408-873d-b0d14d3529b5-maas-api-xks-ca" pod "101c9f6c-b9b6-4408-873d-b0d14d3529b5" (UID: "101c9f6c-b9b6-4408-873d-b0d14d3529b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:12:36.516074 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:36.516025 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8ktz\" (UniqueName: \"kubernetes.io/projected/101c9f6c-b9b6-4408-873d-b0d14d3529b5-kube-api-access-d8ktz\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:36.516074 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:36.516060 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101c9f6c-b9b6-4408-873d-b0d14d3529b5-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:37.116781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:37.116746 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv" event={"ID":"101c9f6c-b9b6-4408-873d-b0d14d3529b5","Type":"ContainerDied","Data":"8ce25c572db4e0fdd98309fcc5e3c3fd2956d219b4c11309412fbe0e82835bca"} Oct 30 02:12:37.116781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:37.116784 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv" Oct 30 02:12:37.117251 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:37.116803 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ce25c572db4e0fdd98309fcc5e3c3fd2956d219b4c11309412fbe0e82835bca" Oct 30 02:12:37.177911 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:37.177874 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:37.286054 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:37.286022 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:38.910539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:38.910504 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:40.599675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:40.599626 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:42.110617 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:42.110584 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:43.685041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:43.685003 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:45.348951 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:45.348913 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:45.486258 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:45.486217 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96"] Oct 30 02:12:45.514144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:45.509931 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:46.957481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:46.957450 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:48.586495 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:48.586430 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:50.200802 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.200763 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:50.408180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.408143 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:50.626203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.626108 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf"] Oct 30 02:12:50.626705 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.626689 5998 cpu_manager.go:414] "containerMap: removing container" podUID="101c9f6c-b9b6-4408-873d-b0d14d3529b5" containerName="revoke-keys" Oct 30 02:12:50.626784 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.626705 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101c9f6c-b9b6-4408-873d-b0d14d3529b5" containerName="revoke-keys" Oct 30 02:12:50.626842 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.626804 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="101c9f6c-b9b6-4408-873d-b0d14d3529b5" containerName="revoke-keys" Oct 30 02:12:50.630303 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.630280 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf" Oct 30 02:12:50.643377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.643348 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:12:50.653759 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.653729 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:12:50.653936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.653776 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:12:50.653936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.653791 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:12:50.658135 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.658110 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf"] Oct 30 02:12:50.731193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.731157 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r56ck\" (UniqueName: \"kubernetes.io/projected/677420bb-8553-4ae9-badd-fe881efddaf4-kube-api-access-r56ck\") pod \"maas-api-revoke-sub-f43081d9-g25sf\" (UID: \"677420bb-8553-4ae9-badd-fe881efddaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf" Oct 30 02:12:50.731396 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.731340 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677420bb-8553-4ae9-badd-fe881efddaf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f43081d9-g25sf\" (UID: \"677420bb-8553-4ae9-badd-fe881efddaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf" Oct 30 02:12:50.731484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.731454 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677420bb-8553-4ae9-badd-fe881efddaf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f43081d9-g25sf\" (UID: \"677420bb-8553-4ae9-badd-fe881efddaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf" Oct 30 02:12:50.833019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.832974 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r56ck\" (UniqueName: \"kubernetes.io/projected/677420bb-8553-4ae9-badd-fe881efddaf4-kube-api-access-r56ck\") pod \"maas-api-revoke-sub-f43081d9-g25sf\" (UID: \"677420bb-8553-4ae9-badd-fe881efddaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf" Oct 30 02:12:50.833243 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.833141 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677420bb-8553-4ae9-badd-fe881efddaf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f43081d9-g25sf\" (UID: \"677420bb-8553-4ae9-badd-fe881efddaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf" Oct 30 02:12:50.833344 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.833246 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677420bb-8553-4ae9-badd-fe881efddaf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f43081d9-g25sf\" (UID: \"677420bb-8553-4ae9-badd-fe881efddaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf" Oct 30 02:12:50.833962 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.833933 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677420bb-8553-4ae9-badd-fe881efddaf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f43081d9-g25sf\" (UID: \"677420bb-8553-4ae9-badd-fe881efddaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf" Oct 30 02:12:50.835927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.835902 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677420bb-8553-4ae9-badd-fe881efddaf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f43081d9-g25sf\" (UID: \"677420bb-8553-4ae9-badd-fe881efddaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf" Oct 30 02:12:50.855059 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.855029 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r56ck\" (UniqueName: \"kubernetes.io/projected/677420bb-8553-4ae9-badd-fe881efddaf4-kube-api-access-r56ck\") pod \"maas-api-revoke-sub-f43081d9-g25sf\" (UID: \"677420bb-8553-4ae9-badd-fe881efddaf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf" Oct 30 02:12:50.944692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:50.944659 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf" Oct 30 02:12:51.069948 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:51.069915 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt"] Oct 30 02:12:51.070172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:51.070137 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" podUID="7d2123b7-6e81-4dce-8893-29ec86f7bd7e" containerName="istio-proxy" containerID="cri-o://1877cd29a4d898fa9bdb654f7b7186e6a62638bfcaf99503336cc1ae993c0470" gracePeriod=30 Oct 30 02:12:51.075641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:51.075609 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww"] Oct 30 02:12:51.075863 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:51.075832 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" podUID="91229273-99e3-4f69-b5d0-ce56b76e309f" containerName="istio-proxy" containerID="cri-o://629ece0768001b01e69ca3f72de5701a7c2c7a5d553ea8e9ca2712d7e42d85ab" gracePeriod=30 Oct 30 02:12:51.086710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:51.081869 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j"] Oct 30 02:12:51.086710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:51.082138 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" podUID="fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" containerName="istio-proxy" containerID="cri-o://85188ad811011e0c89f1a735ad5b702013f9c8cd8d09aa6d5ff59e30e3a0e464" gracePeriod=30 Oct 30 02:12:51.087348 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:51.087323 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q"] Oct 30 02:12:51.087559 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:51.087529 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" podUID="f935c19c-9ed9-4f78-aa0a-d52823a3de7b" containerName="istio-proxy" containerID="cri-o://65e0034f3432e095a1dcda39a7952727afee1c00b78b680ff5d14b191dbba77b" gracePeriod=30 Oct 30 02:12:51.278360 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:12:51.278329 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677420bb_8553_4ae9_badd_fe881efddaf4.slice/crio-8f92801fd420e23b83ff4f2b4df7dd166f4808db154671af63e1b18b8389967a WatchSource:0}: Error finding container 8f92801fd420e23b83ff4f2b4df7dd166f4808db154671af63e1b18b8389967a: Status 404 returned error can't find the container with id 8f92801fd420e23b83ff4f2b4df7dd166f4808db154671af63e1b18b8389967a Oct 30 02:12:51.280139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:51.280111 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf"] Oct 30 02:12:51.882331 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:51.882249 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:52.252366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:52.252325 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf" event={"ID":"677420bb-8553-4ae9-badd-fe881efddaf4","Type":"ContainerStarted","Data":"3c24b3088408dffbcad7e6cb0d3a2488c057525796291d43b1b3a267b2cba981"} Oct 30 02:12:52.252564 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:52.252398 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf" event={"ID":"677420bb-8553-4ae9-badd-fe881efddaf4","Type":"ContainerStarted","Data":"8f92801fd420e23b83ff4f2b4df7dd166f4808db154671af63e1b18b8389967a"} Oct 30 02:12:52.297919 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:52.297864 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf" podStartSLOduration=2.297849056 podStartE2EDuration="2.297849056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:12:52.294524268 +0000 UTC m=+2601.873315349" watchObservedRunningTime="2026-10-30 02:12:52.297849056 +0000 UTC m=+2601.876640128" Oct 30 02:12:53.261665 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:53.261632 5998 generic.go:356] "Generic (PLEG): container finished" podID="677420bb-8553-4ae9-badd-fe881efddaf4" containerID="3c24b3088408dffbcad7e6cb0d3a2488c057525796291d43b1b3a267b2cba981" exitCode=0 Oct 30 02:12:53.261864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:53.261711 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf" event={"ID":"677420bb-8553-4ae9-badd-fe881efddaf4","Type":"ContainerDied","Data":"3c24b3088408dffbcad7e6cb0d3a2488c057525796291d43b1b3a267b2cba981"} Oct 30 02:12:53.473392 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:53.473354 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b"] Oct 30 02:12:53.500805 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:53.498606 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:53.607170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:53.606525 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:54.014471 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:54.014432 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:54.419766 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:54.419729 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:54.479233 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:54.479194 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf" Oct 30 02:12:54.598803 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:54.598698 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/677420bb-8553-4ae9-badd-fe881efddaf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677420bb-8553-4ae9-badd-fe881efddaf4-maas-api-openshift-ca\") pod \"677420bb-8553-4ae9-badd-fe881efddaf4\" (UID: \"677420bb-8553-4ae9-badd-fe881efddaf4\") " Oct 30 02:12:54.598803 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:54.598789 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/677420bb-8553-4ae9-badd-fe881efddaf4-kube-api-access-r56ck\" (UniqueName: \"kubernetes.io/projected/677420bb-8553-4ae9-badd-fe881efddaf4-kube-api-access-r56ck\") pod \"677420bb-8553-4ae9-badd-fe881efddaf4\" (UID: \"677420bb-8553-4ae9-badd-fe881efddaf4\") " Oct 30 02:12:54.599072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:54.599019 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/677420bb-8553-4ae9-badd-fe881efddaf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677420bb-8553-4ae9-badd-fe881efddaf4-maas-api-xks-ca\") pod \"677420bb-8553-4ae9-badd-fe881efddaf4\" (UID: \"677420bb-8553-4ae9-badd-fe881efddaf4\") " Oct 30 02:12:54.599072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:54.599053 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/677420bb-8553-4ae9-badd-fe881efddaf4-maas-api-openshift-ca" pod "677420bb-8553-4ae9-badd-fe881efddaf4" (UID: "677420bb-8553-4ae9-badd-fe881efddaf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:12:54.599716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:54.599687 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677420bb-8553-4ae9-badd-fe881efddaf4-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:54.601434 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:54.601410 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677420bb-8553-4ae9-badd-fe881efddaf4-kube-api-access-r56ck" pod "677420bb-8553-4ae9-badd-fe881efddaf4" (UID: "677420bb-8553-4ae9-badd-fe881efddaf4"). InnerVolumeSpecName "kube-api-access-r56ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:54.601679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:54.601614 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677420bb-8553-4ae9-badd-fe881efddaf4-maas-api-xks-ca" pod "677420bb-8553-4ae9-badd-fe881efddaf4" (UID: "677420bb-8553-4ae9-badd-fe881efddaf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:12:54.700896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:54.700856 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r56ck\" (UniqueName: \"kubernetes.io/projected/677420bb-8553-4ae9-badd-fe881efddaf4-kube-api-access-r56ck\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:54.700896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:54.700896 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677420bb-8553-4ae9-badd-fe881efddaf4-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:55.220196 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:12:55.220152 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:12:55.277920 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:55.277884 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:55.281434 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:55.281405 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf" Oct 30 02:12:55.281633 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:55.281404 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf" event={"ID":"677420bb-8553-4ae9-badd-fe881efddaf4","Type":"ContainerDied","Data":"8f92801fd420e23b83ff4f2b4df7dd166f4808db154671af63e1b18b8389967a"} Oct 30 02:12:55.281633 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:55.281535 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f92801fd420e23b83ff4f2b4df7dd166f4808db154671af63e1b18b8389967a" Oct 30 02:12:55.398115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:55.396708 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:12:55.398115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:55.396788 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" podUID="7d2123b7-6e81-4dce-8893-29ec86f7bd7e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:12:55.751182 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:55.751141 5998 patch_prober.go:29] interesting pod/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:12:55.751694 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:55.751198 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" podUID="91229273-99e3-4f69-b5d0-ce56b76e309f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:12:56.342850 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.341225 5998 generic.go:356] "Generic (PLEG): container finished" podID="7d2123b7-6e81-4dce-8893-29ec86f7bd7e" containerID="1877cd29a4d898fa9bdb654f7b7186e6a62638bfcaf99503336cc1ae993c0470" exitCode=0 Oct 30 02:12:56.342850 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.341498 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" event={"ID":"7d2123b7-6e81-4dce-8893-29ec86f7bd7e","Type":"ContainerDied","Data":"1877cd29a4d898fa9bdb654f7b7186e6a62638bfcaf99503336cc1ae993c0470"} Oct 30 02:12:56.348511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.347601 5998 generic.go:356] "Generic (PLEG): container finished" podID="91229273-99e3-4f69-b5d0-ce56b76e309f" containerID="629ece0768001b01e69ca3f72de5701a7c2c7a5d553ea8e9ca2712d7e42d85ab" exitCode=0 Oct 30 02:12:56.348511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.347701 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" event={"ID":"91229273-99e3-4f69-b5d0-ce56b76e309f","Type":"ContainerDied","Data":"629ece0768001b01e69ca3f72de5701a7c2c7a5d553ea8e9ca2712d7e42d85ab"} Oct 30 02:12:56.351917 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.351577 5998 generic.go:356] "Generic (PLEG): container finished" podID="f935c19c-9ed9-4f78-aa0a-d52823a3de7b" containerID="65e0034f3432e095a1dcda39a7952727afee1c00b78b680ff5d14b191dbba77b" exitCode=0 Oct 30 02:12:56.351917 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.351673 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" event={"ID":"f935c19c-9ed9-4f78-aa0a-d52823a3de7b","Type":"ContainerDied","Data":"65e0034f3432e095a1dcda39a7952727afee1c00b78b680ff5d14b191dbba77b"} Oct 30 02:12:56.354211 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.354182 5998 generic.go:356] "Generic (PLEG): container finished" podID="fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" containerID="85188ad811011e0c89f1a735ad5b702013f9c8cd8d09aa6d5ff59e30e3a0e464" exitCode=0 Oct 30 02:12:56.354359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.354224 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" event={"ID":"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3","Type":"ContainerDied","Data":"85188ad811011e0c89f1a735ad5b702013f9c8cd8d09aa6d5ff59e30e3a0e464"} Oct 30 02:12:56.428110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.427316 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 02:12:56.462951 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.462923 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:12:56.472800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.472779 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:12:56.530481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.530433 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-envoy\") pod \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " Oct 30 02:12:56.530676 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.530556 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-token\" (UniqueName: \"kubernetes.io/projected/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-token\") pod \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " Oct 30 02:12:56.530676 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.530666 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-workload-socket\") pod \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " Oct 30 02:12:56.530793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.530701 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-kube-api-access-d2wgh\" (UniqueName: \"kubernetes.io/projected/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-kube-api-access-d2wgh\") pod \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " Oct 30 02:12:56.530793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.530749 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istiod-ca-cert\") pod \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " Oct 30 02:12:56.530793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.530793 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-podinfo\") pod \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " Oct 30 02:12:56.530987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.530827 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-credential-socket\") pod \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " Oct 30 02:12:56.530987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.530934 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-workload-certs\") pod \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " Oct 30 02:12:56.531111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.530983 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-workload-socket" pod "7d2123b7-6e81-4dce-8893-29ec86f7bd7e" (UID: "7d2123b7-6e81-4dce-8893-29ec86f7bd7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.531111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.531012 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-data\") pod \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\" (UID: \"7d2123b7-6e81-4dce-8893-29ec86f7bd7e\") " Oct 30 02:12:56.531594 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.531568 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.531831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.531773 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-data" pod "7d2123b7-6e81-4dce-8893-29ec86f7bd7e" (UID: "7d2123b7-6e81-4dce-8893-29ec86f7bd7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.532828 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.532787 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-workload-certs" pod "7d2123b7-6e81-4dce-8893-29ec86f7bd7e" (UID: "7d2123b7-6e81-4dce-8893-29ec86f7bd7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.532828 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.532801 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-credential-socket" pod "7d2123b7-6e81-4dce-8893-29ec86f7bd7e" (UID: "7d2123b7-6e81-4dce-8893-29ec86f7bd7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.533289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.533262 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istiod-ca-cert" pod "7d2123b7-6e81-4dce-8893-29ec86f7bd7e" (UID: "7d2123b7-6e81-4dce-8893-29ec86f7bd7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:12:56.544773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.544741 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-envoy" pod "7d2123b7-6e81-4dce-8893-29ec86f7bd7e" (UID: "7d2123b7-6e81-4dce-8893-29ec86f7bd7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.546931 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.546902 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-kube-api-access-d2wgh" pod "7d2123b7-6e81-4dce-8893-29ec86f7bd7e" (UID: "7d2123b7-6e81-4dce-8893-29ec86f7bd7e"). InnerVolumeSpecName "kube-api-access-d2wgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:56.547123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.547057 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-token" pod "7d2123b7-6e81-4dce-8893-29ec86f7bd7e" (UID: "7d2123b7-6e81-4dce-8893-29ec86f7bd7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:56.547458 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.547434 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-podinfo" pod "7d2123b7-6e81-4dce-8893-29ec86f7bd7e" (UID: "7d2123b7-6e81-4dce-8893-29ec86f7bd7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:12:56.632435 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.632342 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-token\" (UniqueName: \"kubernetes.io/projected/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-token\") pod \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " Oct 30 02:12:56.632435 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.632412 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-envoy\") pod \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " Oct 30 02:12:56.632667 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.632473 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-workload-socket\") pod \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " Oct 30 02:12:56.632667 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.632523 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istiod-ca-cert\") pod \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " Oct 30 02:12:56.632667 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.632601 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-data\") pod \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " Oct 30 02:12:56.632667 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.632643 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-credential-socket\") pod \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " Oct 30 02:12:56.632895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.632683 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-workload-certs\") pod \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " Oct 30 02:12:56.632895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.632721 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-workload-certs\") pod \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " Oct 30 02:12:56.632895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.632758 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-podinfo\") pod \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " Oct 30 02:12:56.632895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.632789 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-podinfo\") pod \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " Oct 30 02:12:56.632895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.632869 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-workload-socket\") pod \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " Oct 30 02:12:56.633309 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.632920 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-envoy\") pod \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " Oct 30 02:12:56.633309 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.632973 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-data\") pod \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " Oct 30 02:12:56.633309 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.633001 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-kube-api-access-m72w5\" (UniqueName: \"kubernetes.io/projected/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-kube-api-access-m72w5\") pod \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " Oct 30 02:12:56.633309 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.633037 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-kube-api-access-f7src\" (UniqueName: \"kubernetes.io/projected/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-kube-api-access-f7src\") pod \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\" (UID: \"f935c19c-9ed9-4f78-aa0a-d52823a3de7b\") " Oct 30 02:12:56.633309 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.633073 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istiod-ca-cert\") pod \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " Oct 30 02:12:56.633309 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.633151 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istiod-ca-cert" pod "f935c19c-9ed9-4f78-aa0a-d52823a3de7b" (UID: "f935c19c-9ed9-4f78-aa0a-d52823a3de7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:12:56.634004 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.633328 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-workload-socket" pod "f935c19c-9ed9-4f78-aa0a-d52823a3de7b" (UID: "f935c19c-9ed9-4f78-aa0a-d52823a3de7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.634004 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.633400 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-token\" (UniqueName: \"kubernetes.io/projected/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-token\") pod \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " Oct 30 02:12:56.634004 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.633468 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-credential-socket\") pod \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\" (UID: \"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3\") " Oct 30 02:12:56.634004 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.633959 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istiod-ca-cert" pod "fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" (UID: "fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:12:56.634269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.634198 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-workload-certs" pod "fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" (UID: "fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.634367 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.634347 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.634431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.634368 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.634431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.634377 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.634431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.634386 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.634431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.634394 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.634431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.634402 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2wgh\" (UniqueName: \"kubernetes.io/projected/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-kube-api-access-d2wgh\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.634431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.634413 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.634431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.634421 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.634431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.634429 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.634874 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.634431 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-credential-socket" pod "f935c19c-9ed9-4f78-aa0a-d52823a3de7b" (UID: "f935c19c-9ed9-4f78-aa0a-d52823a3de7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.634874 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.634439 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.634874 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.634482 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.634874 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.634498 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d2123b7-6e81-4dce-8893-29ec86f7bd7e-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.635137 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.635023 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-data" pod "f935c19c-9ed9-4f78-aa0a-d52823a3de7b" (UID: "f935c19c-9ed9-4f78-aa0a-d52823a3de7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.635490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.635470 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-workload-certs" pod "f935c19c-9ed9-4f78-aa0a-d52823a3de7b" (UID: "f935c19c-9ed9-4f78-aa0a-d52823a3de7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.635640 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.635622 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-workload-socket" pod "fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" (UID: "fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.635918 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.635891 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-data" pod "fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" (UID: "fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.636905 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.636459 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-credential-socket" pod "fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" (UID: "fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.636905 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.636577 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-envoy" pod "fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" (UID: "fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.636905 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.636743 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-podinfo" pod "f935c19c-9ed9-4f78-aa0a-d52823a3de7b" (UID: "f935c19c-9ed9-4f78-aa0a-d52823a3de7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:12:56.637147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.636947 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-token" pod "fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" (UID: "fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:56.637147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.637030 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-token" pod "f935c19c-9ed9-4f78-aa0a-d52823a3de7b" (UID: "f935c19c-9ed9-4f78-aa0a-d52823a3de7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:56.637804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.637780 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-kube-api-access-m72w5" pod "fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" (UID: "fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3"). InnerVolumeSpecName "kube-api-access-m72w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:56.638929 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.638848 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-envoy" pod "f935c19c-9ed9-4f78-aa0a-d52823a3de7b" (UID: "f935c19c-9ed9-4f78-aa0a-d52823a3de7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.638929 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.638908 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-podinfo" pod "fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" (UID: "fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:12:56.639049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.639026 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-kube-api-access-f7src" pod "f935c19c-9ed9-4f78-aa0a-d52823a3de7b" (UID: "f935c19c-9ed9-4f78-aa0a-d52823a3de7b"). InnerVolumeSpecName "kube-api-access-f7src". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:56.735997 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.735965 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.735997 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.735995 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.736213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.736009 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.736213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.736021 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.736213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.736032 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.736213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.736043 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.736213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.736055 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.736213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.736068 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.736213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.736080 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.736213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.736110 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.736213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.736122 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.736213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.736134 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m72w5\" (UniqueName: \"kubernetes.io/projected/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-kube-api-access-m72w5\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.736213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.736148 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7src\" (UniqueName: \"kubernetes.io/projected/f935c19c-9ed9-4f78-aa0a-d52823a3de7b-kube-api-access-f7src\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.736213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.736171 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.742309 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.742285 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 02:12:56.837636 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.837594 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-workload-certs\") pod \"91229273-99e3-4f69-b5d0-ce56b76e309f\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " Oct 30 02:12:56.838155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.837667 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-envoy\") pod \"91229273-99e3-4f69-b5d0-ce56b76e309f\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " Oct 30 02:12:56.838155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.837725 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91229273-99e3-4f69-b5d0-ce56b76e309f-kube-api-access-dw88g\" (UniqueName: \"kubernetes.io/projected/91229273-99e3-4f69-b5d0-ce56b76e309f-kube-api-access-dw88g\") pod \"91229273-99e3-4f69-b5d0-ce56b76e309f\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " Oct 30 02:12:56.838155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.837868 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-token\" (UniqueName: \"kubernetes.io/projected/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-token\") pod \"91229273-99e3-4f69-b5d0-ce56b76e309f\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " Oct 30 02:12:56.838155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.837921 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-workload-socket\") pod \"91229273-99e3-4f69-b5d0-ce56b76e309f\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " Oct 30 02:12:56.838155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.837984 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-workload-certs" pod "91229273-99e3-4f69-b5d0-ce56b76e309f" (UID: "91229273-99e3-4f69-b5d0-ce56b76e309f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.838155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.838003 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-credential-socket\") pod \"91229273-99e3-4f69-b5d0-ce56b76e309f\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " Oct 30 02:12:56.838155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.838085 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-podinfo\") pod \"91229273-99e3-4f69-b5d0-ce56b76e309f\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " Oct 30 02:12:56.838589 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.838244 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91229273-99e3-4f69-b5d0-ce56b76e309f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91229273-99e3-4f69-b5d0-ce56b76e309f-istiod-ca-cert\") pod \"91229273-99e3-4f69-b5d0-ce56b76e309f\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " Oct 30 02:12:56.838589 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.838259 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-credential-socket" pod "91229273-99e3-4f69-b5d0-ce56b76e309f" (UID: "91229273-99e3-4f69-b5d0-ce56b76e309f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.838589 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.838275 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-data\") pod \"91229273-99e3-4f69-b5d0-ce56b76e309f\" (UID: \"91229273-99e3-4f69-b5d0-ce56b76e309f\") " Oct 30 02:12:56.838589 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.838510 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-workload-socket" pod "91229273-99e3-4f69-b5d0-ce56b76e309f" (UID: "91229273-99e3-4f69-b5d0-ce56b76e309f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.838807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.838720 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-data" pod "91229273-99e3-4f69-b5d0-ce56b76e309f" (UID: "91229273-99e3-4f69-b5d0-ce56b76e309f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.838869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.838817 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91229273-99e3-4f69-b5d0-ce56b76e309f-istiod-ca-cert" pod "91229273-99e3-4f69-b5d0-ce56b76e309f" (UID: "91229273-99e3-4f69-b5d0-ce56b76e309f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:12:56.839167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.839148 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.839167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.839168 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.839327 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.839178 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.839327 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.839191 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.839327 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.839206 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91229273-99e3-4f69-b5d0-ce56b76e309f-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.840398 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.840376 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91229273-99e3-4f69-b5d0-ce56b76e309f-kube-api-access-dw88g" pod "91229273-99e3-4f69-b5d0-ce56b76e309f" (UID: "91229273-99e3-4f69-b5d0-ce56b76e309f"). InnerVolumeSpecName "kube-api-access-dw88g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:56.840530 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.840433 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-envoy" pod "91229273-99e3-4f69-b5d0-ce56b76e309f" (UID: "91229273-99e3-4f69-b5d0-ce56b76e309f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:56.840530 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.840484 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-podinfo" pod "91229273-99e3-4f69-b5d0-ce56b76e309f" (UID: "91229273-99e3-4f69-b5d0-ce56b76e309f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:12:56.840787 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.840766 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-token" pod "91229273-99e3-4f69-b5d0-ce56b76e309f" (UID: "91229273-99e3-4f69-b5d0-ce56b76e309f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:56.940494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.940461 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.940494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.940489 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw88g\" (UniqueName: \"kubernetes.io/projected/91229273-99e3-4f69-b5d0-ce56b76e309f-kube-api-access-dw88g\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.940494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.940498 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.940786 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.940506 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91229273-99e3-4f69-b5d0-ce56b76e309f-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:12:56.971552 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:56.971517 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:57.363721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.363625 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" event={"ID":"7d2123b7-6e81-4dce-8893-29ec86f7bd7e","Type":"ContainerDied","Data":"a506192fa53bf73cdce56952358a734d810377903e5fc3b4a20d58ae28fe8f91"} Oct 30 02:12:57.363721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.363702 5998 scope.go:122] "RemoveContainer" containerID="1877cd29a4d898fa9bdb654f7b7186e6a62638bfcaf99503336cc1ae993c0470" Oct 30 02:12:57.363969 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.363645 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt" Oct 30 02:12:57.365467 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.365448 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" Oct 30 02:12:57.365614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.365467 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww" event={"ID":"91229273-99e3-4f69-b5d0-ce56b76e309f","Type":"ContainerDied","Data":"bbc52f5c510eea71d02625b3941395f784c1568c1fbb9cd2f7188389f972dba7"} Oct 30 02:12:57.372200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.371626 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" event={"ID":"f935c19c-9ed9-4f78-aa0a-d52823a3de7b","Type":"ContainerDied","Data":"880d5d3111a76ebbfa593394824a8d7f624b484d99462acd489c1da4b8341651"} Oct 30 02:12:57.372200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.371737 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q" Oct 30 02:12:57.376649 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.376631 5998 scope.go:122] "RemoveContainer" containerID="629ece0768001b01e69ca3f72de5701a7c2c7a5d553ea8e9ca2712d7e42d85ab" Oct 30 02:12:57.377559 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.377431 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" Oct 30 02:12:57.377559 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.377465 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j" event={"ID":"fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3","Type":"ContainerDied","Data":"d5aef54cd74d9b5e352acbb97473e52e01e4bd20df8c42e980e4613959112867"} Oct 30 02:12:57.394794 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.394775 5998 scope.go:122] "RemoveContainer" containerID="65e0034f3432e095a1dcda39a7952727afee1c00b78b680ff5d14b191dbba77b" Oct 30 02:12:57.405005 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.404985 5998 scope.go:122] "RemoveContainer" containerID="85188ad811011e0c89f1a735ad5b702013f9c8cd8d09aa6d5ff59e30e3a0e464" Oct 30 02:12:57.457295 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.457264 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q"] Oct 30 02:12:57.467332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.467308 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-22z9q"] Oct 30 02:12:57.494416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.494388 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j"] Oct 30 02:12:57.498069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.498042 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-hr59j"] Oct 30 02:12:57.530616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.530588 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt"] Oct 30 02:12:57.534234 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.534209 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-fdwmt"] Oct 30 02:12:57.562635 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.562600 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww"] Oct 30 02:12:57.565822 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.565793 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c40a89-openshift-default-6dfbd8dcb7-p9rww"] Oct 30 02:12:57.806259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.806218 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:57.918967 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:57.918933 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:58.486299 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.486262 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11196-6de19d"] Oct 30 02:12:58.486814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.486799 5998 cpu_manager.go:414] "containerMap: removing container" podUID="fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" containerName="istio-proxy" Oct 30 02:12:58.486862 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.486815 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" containerName="istio-proxy" Oct 30 02:12:58.486862 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.486832 5998 cpu_manager.go:414] "containerMap: removing container" podUID="f935c19c-9ed9-4f78-aa0a-d52823a3de7b" containerName="istio-proxy" Oct 30 02:12:58.486862 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.486837 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f935c19c-9ed9-4f78-aa0a-d52823a3de7b" containerName="istio-proxy" Oct 30 02:12:58.486862 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.486848 5998 cpu_manager.go:414] "containerMap: removing container" podUID="677420bb-8553-4ae9-badd-fe881efddaf4" containerName="revoke-keys" Oct 30 02:12:58.486862 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.486853 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="677420bb-8553-4ae9-badd-fe881efddaf4" containerName="revoke-keys" Oct 30 02:12:58.487022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.486879 5998 cpu_manager.go:414] "containerMap: removing container" podUID="91229273-99e3-4f69-b5d0-ce56b76e309f" containerName="istio-proxy" Oct 30 02:12:58.487022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.486885 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91229273-99e3-4f69-b5d0-ce56b76e309f" containerName="istio-proxy" Oct 30 02:12:58.487022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.486904 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7d2123b7-6e81-4dce-8893-29ec86f7bd7e" containerName="istio-proxy" Oct 30 02:12:58.487022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.486910 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2123b7-6e81-4dce-8893-29ec86f7bd7e" containerName="istio-proxy" Oct 30 02:12:58.487022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.487017 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="f935c19c-9ed9-4f78-aa0a-d52823a3de7b" containerName="istio-proxy" Oct 30 02:12:58.487198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.487029 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="677420bb-8553-4ae9-badd-fe881efddaf4" containerName="revoke-keys" Oct 30 02:12:58.487198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.487040 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="91229273-99e3-4f69-b5d0-ce56b76e309f" containerName="istio-proxy" Oct 30 02:12:58.487198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.487052 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2123b7-6e81-4dce-8893-29ec86f7bd7e" containerName="istio-proxy" Oct 30 02:12:58.487198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.487061 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" containerName="istio-proxy" Oct 30 02:12:58.490219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.490201 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-6de19d" Oct 30 02:12:58.520996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.520966 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11196-6de19d"] Oct 30 02:12:58.525751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.525724 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2sddb\"" Oct 30 02:12:58.546723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.546685 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2123b7-6e81-4dce-8893-29ec86f7bd7e" path="/var/lib/kubelet/pods/7d2123b7-6e81-4dce-8893-29ec86f7bd7e/volumes" Oct 30 02:12:58.547504 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.547482 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91229273-99e3-4f69-b5d0-ce56b76e309f" path="/var/lib/kubelet/pods/91229273-99e3-4f69-b5d0-ce56b76e309f/volumes" Oct 30 02:12:58.548134 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.548115 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f935c19c-9ed9-4f78-aa0a-d52823a3de7b" path="/var/lib/kubelet/pods/f935c19c-9ed9-4f78-aa0a-d52823a3de7b/volumes" Oct 30 02:12:58.549249 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.549229 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3" path="/var/lib/kubelet/pods/fd00e78e-d69e-4a6c-99a4-ba99c76cc5b3/volumes" Oct 30 02:12:58.667604 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.667564 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6zsk\" (UniqueName: \"kubernetes.io/projected/af6aaee0-66bf-4f36-aa84-5c0710048bc3-kube-api-access-s6zsk\") pod \"test-curl-11196-6de19d\" (UID: \"af6aaee0-66bf-4f36-aa84-5c0710048bc3\") " pod="opendatahub/test-curl-11196-6de19d" Oct 30 02:12:58.768729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.768639 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6zsk\" (UniqueName: \"kubernetes.io/projected/af6aaee0-66bf-4f36-aa84-5c0710048bc3-kube-api-access-s6zsk\") pod \"test-curl-11196-6de19d\" (UID: \"af6aaee0-66bf-4f36-aa84-5c0710048bc3\") " pod="opendatahub/test-curl-11196-6de19d" Oct 30 02:12:58.790355 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.790318 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6zsk\" (UniqueName: \"kubernetes.io/projected/af6aaee0-66bf-4f36-aa84-5c0710048bc3-kube-api-access-s6zsk\") pod \"test-curl-11196-6de19d\" (UID: \"af6aaee0-66bf-4f36-aa84-5c0710048bc3\") " pod="opendatahub/test-curl-11196-6de19d" Oct 30 02:12:58.802336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.802304 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-6de19d" Oct 30 02:12:58.944455 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:58.944417 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:59.108835 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:59.108738 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11196-6de19d"] Oct 30 02:12:59.114898 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:12:59.114850 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf6aaee0_66bf_4f36_aa84_5c0710048bc3.slice/crio-f5610d247282cd460df831afec3d61b3d76889c3af4d82159c83812c5431748c WatchSource:0}: Error finding container f5610d247282cd460df831afec3d61b3d76889c3af4d82159c83812c5431748c: Status 404 returned error can't find the container with id f5610d247282cd460df831afec3d61b3d76889c3af4d82159c83812c5431748c Oct 30 02:12:59.403558 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:59.403458 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11196-6de19d" event={"ID":"af6aaee0-66bf-4f36-aa84-5c0710048bc3","Type":"ContainerStarted","Data":"e4ee1b85e8eafe694f4bbf4f4978b548855d2c67efc2b61de48de26150b9a4ad"} Oct 30 02:12:59.403558 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:59.403503 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11196-6de19d" event={"ID":"af6aaee0-66bf-4f36-aa84-5c0710048bc3","Type":"ContainerStarted","Data":"f5610d247282cd460df831afec3d61b3d76889c3af4d82159c83812c5431748c"} Oct 30 02:12:59.439475 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:59.439422 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11196-6de19d" podStartSLOduration=1.439408188 podStartE2EDuration="1.439408188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:12:59.438611256 +0000 UTC m=+2609.017402336" watchObservedRunningTime="2026-10-30 02:12:59.439408188 +0000 UTC m=+2609.018199270" Oct 30 02:12:59.535560 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:59.535514 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:12:59.648327 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:12:59.648292 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:00.312315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.310728 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11196-6de19d"] Oct 30 02:13:00.320400 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.320367 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11196-6de19d"] Oct 30 02:13:00.470466 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.470428 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11196-532d8a"] Oct 30 02:13:00.471072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.471057 5998 cpu_manager.go:414] "containerMap: removing container" podUID="af6aaee0-66bf-4f36-aa84-5c0710048bc3" containerName="test-curl-11196-6de19d" Oct 30 02:13:00.471125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.471075 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af6aaee0-66bf-4f36-aa84-5c0710048bc3" containerName="test-curl-11196-6de19d" Oct 30 02:13:00.471192 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.471182 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="af6aaee0-66bf-4f36-aa84-5c0710048bc3" containerName="test-curl-11196-6de19d" Oct 30 02:13:00.474293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.474277 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-532d8a" Oct 30 02:13:00.492891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.492866 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11196-532d8a"] Oct 30 02:13:00.500876 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.500845 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz75w\" (UniqueName: \"kubernetes.io/projected/1455eaa7-24ee-45a2-bb37-155870b1ebf8-kube-api-access-zz75w\") pod \"test-curl-11196-532d8a\" (UID: \"1455eaa7-24ee-45a2-bb37-155870b1ebf8\") " pod="opendatahub/test-curl-11196-532d8a" Oct 30 02:13:00.517869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.517839 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" podUID="f826281b-5ba7-427b-800b-e8fbb621793a" containerName="main" containerID="cri-o://864321e8d55cb7cf36aabf8ca66cb9ea3137c595c305dcc9574f2242df6c967e" gracePeriod=45 Oct 30 02:13:00.602053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.601964 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz75w\" (UniqueName: \"kubernetes.io/projected/1455eaa7-24ee-45a2-bb37-155870b1ebf8-kube-api-access-zz75w\") pod \"test-curl-11196-532d8a\" (UID: \"1455eaa7-24ee-45a2-bb37-155870b1ebf8\") " pod="opendatahub/test-curl-11196-532d8a" Oct 30 02:13:00.623386 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.623352 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz75w\" (UniqueName: \"kubernetes.io/projected/1455eaa7-24ee-45a2-bb37-155870b1ebf8-kube-api-access-zz75w\") pod \"test-curl-11196-532d8a\" (UID: \"1455eaa7-24ee-45a2-bb37-155870b1ebf8\") " pod="opendatahub/test-curl-11196-532d8a" Oct 30 02:13:00.760732 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.760699 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:00.786963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.786937 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-532d8a" Oct 30 02:13:00.839869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.839833 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:13:00.915519 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.914345 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-model-cache\") pod \"f826281b-5ba7-427b-800b-e8fbb621793a\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " Oct 30 02:13:00.915519 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.914500 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-tmp-dir\") pod \"f826281b-5ba7-427b-800b-e8fbb621793a\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " Oct 30 02:13:00.915519 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.914532 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-home\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-home\") pod \"f826281b-5ba7-427b-800b-e8fbb621793a\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " Oct 30 02:13:00.915519 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.914561 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f826281b-5ba7-427b-800b-e8fbb621793a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f826281b-5ba7-427b-800b-e8fbb621793a-tls-certs\") pod \"f826281b-5ba7-427b-800b-e8fbb621793a\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " Oct 30 02:13:00.915519 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.914590 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-dshm\") pod \"f826281b-5ba7-427b-800b-e8fbb621793a\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " Oct 30 02:13:00.915519 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.914698 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f826281b-5ba7-427b-800b-e8fbb621793a-kube-api-access-kd8f5\" (UniqueName: \"kubernetes.io/projected/f826281b-5ba7-427b-800b-e8fbb621793a-kube-api-access-kd8f5\") pod \"f826281b-5ba7-427b-800b-e8fbb621793a\" (UID: \"f826281b-5ba7-427b-800b-e8fbb621793a\") " Oct 30 02:13:00.915971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.915801 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-home" pod "f826281b-5ba7-427b-800b-e8fbb621793a" (UID: "f826281b-5ba7-427b-800b-e8fbb621793a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:00.916037 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.915985 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-model-cache" pod "f826281b-5ba7-427b-800b-e8fbb621793a" (UID: "f826281b-5ba7-427b-800b-e8fbb621793a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:00.918556 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.916154 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-tmp-dir" pod "f826281b-5ba7-427b-800b-e8fbb621793a" (UID: "f826281b-5ba7-427b-800b-e8fbb621793a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:00.920132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.919217 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f826281b-5ba7-427b-800b-e8fbb621793a-tls-certs" pod "f826281b-5ba7-427b-800b-e8fbb621793a" (UID: "f826281b-5ba7-427b-800b-e8fbb621793a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:13:00.920132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.919371 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-dshm" pod "f826281b-5ba7-427b-800b-e8fbb621793a" (UID: "f826281b-5ba7-427b-800b-e8fbb621793a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:00.921496 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:00.921459 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f826281b-5ba7-427b-800b-e8fbb621793a-kube-api-access-kd8f5" pod "f826281b-5ba7-427b-800b-e8fbb621793a" (UID: "f826281b-5ba7-427b-800b-e8fbb621793a"). InnerVolumeSpecName "kube-api-access-kd8f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:01.016611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.016576 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd8f5\" (UniqueName: \"kubernetes.io/projected/f826281b-5ba7-427b-800b-e8fbb621793a-kube-api-access-kd8f5\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:01.016611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.016607 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:01.016611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.016620 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:01.016911 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.016627 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:01.016911 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.016636 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f826281b-5ba7-427b-800b-e8fbb621793a-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:01.016911 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.016644 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f826281b-5ba7-427b-800b-e8fbb621793a-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:01.026176 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.026136 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d"] Oct 30 02:13:01.026677 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.026662 5998 cpu_manager.go:414] "containerMap: removing container" podUID="f826281b-5ba7-427b-800b-e8fbb621793a" containerName="main" Oct 30 02:13:01.026763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.026677 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f826281b-5ba7-427b-800b-e8fbb621793a" containerName="main" Oct 30 02:13:01.026826 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.026771 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="f826281b-5ba7-427b-800b-e8fbb621793a" containerName="main" Oct 30 02:13:01.030231 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.030208 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d" Oct 30 02:13:01.043121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.043084 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:13:01.043248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.043130 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:13:01.043695 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.043677 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:13:01.043760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.043728 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:13:01.050983 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.050954 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d"] Oct 30 02:13:01.117466 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.117433 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e44e724e-48n4d\" (UID: \"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d" Oct 30 02:13:01.117466 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.117475 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e44e724e-48n4d\" (UID: \"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d" Oct 30 02:13:01.117713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.117515 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpx42\" (UniqueName: \"kubernetes.io/projected/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-kube-api-access-tpx42\") pod \"maas-api-revoke-sub-e44e724e-48n4d\" (UID: \"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d" Oct 30 02:13:01.139557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.139475 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11196-532d8a"] Oct 30 02:13:01.143904 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:13:01.143857 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1455eaa7_24ee_45a2_bb37_155870b1ebf8.slice/crio-772869363b4eace2dc0cdef1f5b5bf5dad3a50c2bc88dea5c92538cc43df8f5b WatchSource:0}: Error finding container 772869363b4eace2dc0cdef1f5b5bf5dad3a50c2bc88dea5c92538cc43df8f5b: Status 404 returned error can't find the container with id 772869363b4eace2dc0cdef1f5b5bf5dad3a50c2bc88dea5c92538cc43df8f5b Oct 30 02:13:01.219416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.219221 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e44e724e-48n4d\" (UID: \"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d" Oct 30 02:13:01.219416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.219287 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpx42\" (UniqueName: \"kubernetes.io/projected/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-kube-api-access-tpx42\") pod \"maas-api-revoke-sub-e44e724e-48n4d\" (UID: \"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d" Oct 30 02:13:01.219558 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.219515 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e44e724e-48n4d\" (UID: \"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d" Oct 30 02:13:01.220329 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.220277 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e44e724e-48n4d\" (UID: \"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d" Oct 30 02:13:01.224166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.224139 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e44e724e-48n4d\" (UID: \"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d" Oct 30 02:13:01.241641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.241612 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpx42\" (UniqueName: \"kubernetes.io/projected/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-kube-api-access-tpx42\") pod \"maas-api-revoke-sub-e44e724e-48n4d\" (UID: \"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d" Oct 30 02:13:01.344583 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.344549 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d" Oct 30 02:13:01.426874 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.426836 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11196-532d8a" event={"ID":"1455eaa7-24ee-45a2-bb37-155870b1ebf8","Type":"ContainerStarted","Data":"034353a3cb3c15856ad87c2ea8a22e64a142cdd60a2d9177f8e920de48a6447f"} Oct 30 02:13:01.427154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.427123 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11196-532d8a" event={"ID":"1455eaa7-24ee-45a2-bb37-155870b1ebf8","Type":"ContainerStarted","Data":"772869363b4eace2dc0cdef1f5b5bf5dad3a50c2bc88dea5c92538cc43df8f5b"} Oct 30 02:13:01.429855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.429825 5998 generic.go:356] "Generic (PLEG): container finished" podID="f826281b-5ba7-427b-800b-e8fbb621793a" containerID="864321e8d55cb7cf36aabf8ca66cb9ea3137c595c305dcc9574f2242df6c967e" exitCode=0 Oct 30 02:13:01.429978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.429919 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" event={"ID":"f826281b-5ba7-427b-800b-e8fbb621793a","Type":"ContainerDied","Data":"864321e8d55cb7cf36aabf8ca66cb9ea3137c595c305dcc9574f2242df6c967e"} Oct 30 02:13:01.429978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.429963 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" event={"ID":"f826281b-5ba7-427b-800b-e8fbb621793a","Type":"ContainerDied","Data":"2fe24ed4388eb88fbf87463de271207657d26f20fbac888b273fb46d291f7b8f"} Oct 30 02:13:01.430923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.429986 5998 scope.go:122] "RemoveContainer" containerID="864321e8d55cb7cf36aabf8ca66cb9ea3137c595c305dcc9574f2242df6c967e" Oct 30 02:13:01.430923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.430025 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11196-6de19d" podUID="af6aaee0-66bf-4f36-aa84-5c0710048bc3" containerName="test-curl-11196-6de19d" containerID="cri-o://e4ee1b85e8eafe694f4bbf4f4978b548855d2c67efc2b61de48de26150b9a4ad" gracePeriod=2 Oct 30 02:13:01.431600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.431584 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96" Oct 30 02:13:01.448976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.448554 5998 scope.go:122] "RemoveContainer" containerID="864321e8d55cb7cf36aabf8ca66cb9ea3137c595c305dcc9574f2242df6c967e" Oct 30 02:13:01.448976 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:13:01.448887 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864321e8d55cb7cf36aabf8ca66cb9ea3137c595c305dcc9574f2242df6c967e\": container with ID starting with 864321e8d55cb7cf36aabf8ca66cb9ea3137c595c305dcc9574f2242df6c967e not found: ID does not exist" containerID="864321e8d55cb7cf36aabf8ca66cb9ea3137c595c305dcc9574f2242df6c967e" Oct 30 02:13:01.448976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.448921 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864321e8d55cb7cf36aabf8ca66cb9ea3137c595c305dcc9574f2242df6c967e"} err="failed to get container status \"864321e8d55cb7cf36aabf8ca66cb9ea3137c595c305dcc9574f2242df6c967e\": rpc error: code = NotFound desc = could not find container \"864321e8d55cb7cf36aabf8ca66cb9ea3137c595c305dcc9574f2242df6c967e\": container with ID starting with 864321e8d55cb7cf36aabf8ca66cb9ea3137c595c305dcc9574f2242df6c967e not found: ID does not exist" Oct 30 02:13:01.462976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.462865 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11196-532d8a" podStartSLOduration=1.462853365 podStartE2EDuration="1.462853365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:01.460932149 +0000 UTC m=+2611.039723232" watchObservedRunningTime="2026-10-30 02:13:01.462853365 +0000 UTC m=+2611.041644446" Oct 30 02:13:01.496441 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.496408 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96"] Oct 30 02:13:01.502033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.501995 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-4153e5/auth-test-model-4153e5-kserve-748754bc97-nzs96"] Oct 30 02:13:01.666835 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:01.665347 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d"] Oct 30 02:13:01.676981 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:13:01.676941 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12abfb1_c47a_4f71_bb25_ecfcf35bd7d8.slice/crio-7754bae435b61a0362758cd042a6d4c233ec3c581aea6ccfe98d8f1b34e16fdb WatchSource:0}: Error finding container 7754bae435b61a0362758cd042a6d4c233ec3c581aea6ccfe98d8f1b34e16fdb: Status 404 returned error can't find the container with id 7754bae435b61a0362758cd042a6d4c233ec3c581aea6ccfe98d8f1b34e16fdb Oct 30 02:13:02.334277 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:02.334181 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11196-532d8a"] Oct 30 02:13:02.339099 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:02.339042 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11196-532d8a"] Oct 30 02:13:02.440743 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:02.440700 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d" event={"ID":"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8","Type":"ContainerStarted","Data":"c3998c8ccf99ddb02270d84cd65daeb1e2ba44590b354878a011a401e15cfaf8"} Oct 30 02:13:02.441228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:02.440768 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d" event={"ID":"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8","Type":"ContainerStarted","Data":"7754bae435b61a0362758cd042a6d4c233ec3c581aea6ccfe98d8f1b34e16fdb"} Oct 30 02:13:02.470182 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:02.470145 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:02.475481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:02.475452 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11196-bc3f78"] Oct 30 02:13:02.475993 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:02.475979 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1455eaa7-24ee-45a2-bb37-155870b1ebf8" containerName="test-curl-11196-532d8a" Oct 30 02:13:02.476051 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:02.475994 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1455eaa7-24ee-45a2-bb37-155870b1ebf8" containerName="test-curl-11196-532d8a" Oct 30 02:13:02.476113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:02.476085 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1455eaa7-24ee-45a2-bb37-155870b1ebf8" containerName="test-curl-11196-532d8a" Oct 30 02:13:02.479548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:02.479524 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-bc3f78" Oct 30 02:13:02.480248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:02.480073 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d" podStartSLOduration=2.480060382 podStartE2EDuration="2.480060382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:02.477426934 +0000 UTC m=+2612.056218014" watchObservedRunningTime="2026-10-30 02:13:02.480060382 +0000 UTC m=+2612.058851462" Oct 30 02:13:02.498972 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:02.498941 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11196-bc3f78"] Oct 30 02:13:02.546812 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:02.544010 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58p7v\" (UniqueName: \"kubernetes.io/projected/9b58471f-cd08-4ba5-8c1b-89b83997dffb-kube-api-access-58p7v\") pod \"test-curl-11196-bc3f78\" (UID: \"9b58471f-cd08-4ba5-8c1b-89b83997dffb\") " pod="opendatahub/test-curl-11196-bc3f78" Oct 30 02:13:02.559453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:02.559416 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f826281b-5ba7-427b-800b-e8fbb621793a" path="/var/lib/kubelet/pods/f826281b-5ba7-427b-800b-e8fbb621793a/volumes" Oct 30 02:13:02.656962 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:02.656919 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58p7v\" (UniqueName: \"kubernetes.io/projected/9b58471f-cd08-4ba5-8c1b-89b83997dffb-kube-api-access-58p7v\") pod \"test-curl-11196-bc3f78\" (UID: \"9b58471f-cd08-4ba5-8c1b-89b83997dffb\") " pod="opendatahub/test-curl-11196-bc3f78" Oct 30 02:13:02.678449 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:02.678406 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58p7v\" (UniqueName: \"kubernetes.io/projected/9b58471f-cd08-4ba5-8c1b-89b83997dffb-kube-api-access-58p7v\") pod \"test-curl-11196-bc3f78\" (UID: \"9b58471f-cd08-4ba5-8c1b-89b83997dffb\") " pod="opendatahub/test-curl-11196-bc3f78" Oct 30 02:13:02.792904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:02.792868 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-bc3f78" Oct 30 02:13:03.127242 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:03.127209 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11196-bc3f78"] Oct 30 02:13:03.129805 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:13:03.129771 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b58471f_cd08_4ba5_8c1b_89b83997dffb.slice/crio-d716b08b3e7055b5f795aa95415389bc21652ce457948b06462242a118aa890c WatchSource:0}: Error finding container d716b08b3e7055b5f795aa95415389bc21652ce457948b06462242a118aa890c: Status 404 returned error can't find the container with id d716b08b3e7055b5f795aa95415389bc21652ce457948b06462242a118aa890c Oct 30 02:13:03.452404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:03.452364 5998 generic.go:356] "Generic (PLEG): container finished" podID="b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8" containerID="c3998c8ccf99ddb02270d84cd65daeb1e2ba44590b354878a011a401e15cfaf8" exitCode=0 Oct 30 02:13:03.453448 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:03.452443 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d" event={"ID":"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8","Type":"ContainerDied","Data":"c3998c8ccf99ddb02270d84cd65daeb1e2ba44590b354878a011a401e15cfaf8"} Oct 30 02:13:03.454419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:03.454394 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11196-bc3f78" event={"ID":"9b58471f-cd08-4ba5-8c1b-89b83997dffb","Type":"ContainerStarted","Data":"b6532b6837ad200d0667c111701fe75ccbc08ea2e46f510877526c0b5771780f"} Oct 30 02:13:03.454567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:03.454429 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11196-bc3f78" event={"ID":"9b58471f-cd08-4ba5-8c1b-89b83997dffb","Type":"ContainerStarted","Data":"d716b08b3e7055b5f795aa95415389bc21652ce457948b06462242a118aa890c"} Oct 30 02:13:03.454630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:03.454597 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11196-532d8a" podUID="1455eaa7-24ee-45a2-bb37-155870b1ebf8" containerName="test-curl-11196-532d8a" containerID="cri-o://034353a3cb3c15856ad87c2ea8a22e64a142cdd60a2d9177f8e920de48a6447f" gracePeriod=2 Oct 30 02:13:03.528379 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:03.528319 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11196-bc3f78" podStartSLOduration=1.528070013 podStartE2EDuration="1.528070013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:03.525397301 +0000 UTC m=+2613.104188383" watchObservedRunningTime="2026-10-30 02:13:03.528070013 +0000 UTC m=+2613.106861093" Oct 30 02:13:03.612487 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:03.612449 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-6de19d" Oct 30 02:13:03.670889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:03.670844 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af6aaee0-66bf-4f36-aa84-5c0710048bc3-kube-api-access-s6zsk\" (UniqueName: \"kubernetes.io/projected/af6aaee0-66bf-4f36-aa84-5c0710048bc3-kube-api-access-s6zsk\") pod \"af6aaee0-66bf-4f36-aa84-5c0710048bc3\" (UID: \"af6aaee0-66bf-4f36-aa84-5c0710048bc3\") " Oct 30 02:13:03.672905 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:03.672874 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6aaee0-66bf-4f36-aa84-5c0710048bc3-kube-api-access-s6zsk" pod "af6aaee0-66bf-4f36-aa84-5c0710048bc3" (UID: "af6aaee0-66bf-4f36-aa84-5c0710048bc3"). InnerVolumeSpecName "kube-api-access-s6zsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:03.772851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:03.772764 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6zsk\" (UniqueName: \"kubernetes.io/projected/af6aaee0-66bf-4f36-aa84-5c0710048bc3-kube-api-access-s6zsk\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:03.979941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:03.979909 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:04.384851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:04.384808 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11196-bc3f78"] Oct 30 02:13:04.388868 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:04.388836 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11196-bc3f78"] Oct 30 02:13:04.462947 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:04.462909 5998 generic.go:356] "Generic (PLEG): container finished" podID="af6aaee0-66bf-4f36-aa84-5c0710048bc3" containerID="e4ee1b85e8eafe694f4bbf4f4978b548855d2c67efc2b61de48de26150b9a4ad" exitCode=137 Oct 30 02:13:04.463433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:04.462958 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-6de19d" Oct 30 02:13:04.463433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:04.463015 5998 scope.go:122] "RemoveContainer" containerID="e4ee1b85e8eafe694f4bbf4f4978b548855d2c67efc2b61de48de26150b9a4ad" Oct 30 02:13:04.473607 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:04.473589 5998 scope.go:122] "RemoveContainer" containerID="e4ee1b85e8eafe694f4bbf4f4978b548855d2c67efc2b61de48de26150b9a4ad" Oct 30 02:13:04.473906 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:13:04.473887 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ee1b85e8eafe694f4bbf4f4978b548855d2c67efc2b61de48de26150b9a4ad\": container with ID starting with e4ee1b85e8eafe694f4bbf4f4978b548855d2c67efc2b61de48de26150b9a4ad not found: ID does not exist" containerID="e4ee1b85e8eafe694f4bbf4f4978b548855d2c67efc2b61de48de26150b9a4ad" Oct 30 02:13:04.473979 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:04.473915 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ee1b85e8eafe694f4bbf4f4978b548855d2c67efc2b61de48de26150b9a4ad"} err="failed to get container status \"e4ee1b85e8eafe694f4bbf4f4978b548855d2c67efc2b61de48de26150b9a4ad\": rpc error: code = NotFound desc = could not find container \"e4ee1b85e8eafe694f4bbf4f4978b548855d2c67efc2b61de48de26150b9a4ad\": container with ID starting with e4ee1b85e8eafe694f4bbf4f4978b548855d2c67efc2b61de48de26150b9a4ad not found: ID does not exist" Oct 30 02:13:04.484140 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:13:04.484108 5998 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11196-6de19d\" is forbidden: User \"system:node:ip-10-0-130-0.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-0.ec2.internal' and this object" podUID="af6aaee0-66bf-4f36-aa84-5c0710048bc3" pod="opendatahub/test-curl-11196-6de19d" Oct 30 02:13:04.551037 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:04.550992 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af6aaee0-66bf-4f36-aa84-5c0710048bc3" path="/var/lib/kubelet/pods/af6aaee0-66bf-4f36-aa84-5c0710048bc3/volumes" Oct 30 02:13:04.637369 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:04.637296 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d" Oct 30 02:13:04.685477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:04.685442 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-maas-api-xks-ca\") pod \"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8\" (UID: \"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8\") " Oct 30 02:13:04.685680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:04.685569 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-kube-api-access-tpx42\" (UniqueName: \"kubernetes.io/projected/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-kube-api-access-tpx42\") pod \"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8\" (UID: \"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8\") " Oct 30 02:13:04.685743 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:04.685687 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-maas-api-openshift-ca\") pod \"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8\" (UID: \"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8\") " Oct 30 02:13:04.686585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:04.686127 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-maas-api-openshift-ca" pod "b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8" (UID: "b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:04.686585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:04.686555 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:04.688853 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:04.688809 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-kube-api-access-tpx42" pod "b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8" (UID: "b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8"). InnerVolumeSpecName "kube-api-access-tpx42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:04.689246 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:04.689218 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-maas-api-xks-ca" pod "b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8" (UID: "b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:13:04.788508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:04.788475 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpx42\" (UniqueName: \"kubernetes.io/projected/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-kube-api-access-tpx42\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:04.788508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:04.788504 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:05.475210 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:05.475157 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d" Oct 30 02:13:05.475210 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:05.475187 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d" event={"ID":"b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8","Type":"ContainerDied","Data":"7754bae435b61a0362758cd042a6d4c233ec3c581aea6ccfe98d8f1b34e16fdb"} Oct 30 02:13:05.475612 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:05.475267 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7754bae435b61a0362758cd042a6d4c233ec3c581aea6ccfe98d8f1b34e16fdb" Oct 30 02:13:05.475612 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:05.475501 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11196-bc3f78" podUID="9b58471f-cd08-4ba5-8c1b-89b83997dffb" containerName="test-curl-11196-bc3f78" containerID="cri-o://b6532b6837ad200d0667c111701fe75ccbc08ea2e46f510877526c0b5771780f" gracePeriod=2 Oct 30 02:13:05.670920 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:05.670890 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-532d8a" Oct 30 02:13:05.694881 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:05.694835 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:05.802895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:05.802795 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1455eaa7-24ee-45a2-bb37-155870b1ebf8-kube-api-access-zz75w\" (UniqueName: \"kubernetes.io/projected/1455eaa7-24ee-45a2-bb37-155870b1ebf8-kube-api-access-zz75w\") pod \"1455eaa7-24ee-45a2-bb37-155870b1ebf8\" (UID: \"1455eaa7-24ee-45a2-bb37-155870b1ebf8\") " Oct 30 02:13:05.804999 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:05.804972 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1455eaa7-24ee-45a2-bb37-155870b1ebf8-kube-api-access-zz75w" pod "1455eaa7-24ee-45a2-bb37-155870b1ebf8" (UID: "1455eaa7-24ee-45a2-bb37-155870b1ebf8"). InnerVolumeSpecName "kube-api-access-zz75w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:05.904694 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:05.904662 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz75w\" (UniqueName: \"kubernetes.io/projected/1455eaa7-24ee-45a2-bb37-155870b1ebf8-kube-api-access-zz75w\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:05.953021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:05.952985 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:06.353382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.353343 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl"] Oct 30 02:13:06.354192 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.354159 5998 cpu_manager.go:414] "containerMap: removing container" podUID="b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8" containerName="revoke-keys" Oct 30 02:13:06.354192 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.354185 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8" containerName="revoke-keys" Oct 30 02:13:06.354422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.354213 5998 cpu_manager.go:414] "containerMap: removing container" podUID="9b58471f-cd08-4ba5-8c1b-89b83997dffb" containerName="test-curl-11196-bc3f78" Oct 30 02:13:06.354422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.354221 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b58471f-cd08-4ba5-8c1b-89b83997dffb" containerName="test-curl-11196-bc3f78" Oct 30 02:13:06.354422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.354385 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8" containerName="revoke-keys" Oct 30 02:13:06.354422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.354400 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b58471f-cd08-4ba5-8c1b-89b83997dffb" containerName="test-curl-11196-bc3f78" Oct 30 02:13:06.362426 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.362398 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.380856 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.380829 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-58fad4-openshift-default-dockercfg-9twt4\"" Oct 30 02:13:06.393148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.393116 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl"] Oct 30 02:13:06.410216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.410172 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-credential-socket\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.410426 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.410269 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1411791d-2b8f-46e0-8b26-8799563d8c4b-istiod-ca-cert\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.410426 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.410304 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-token\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.410426 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.410341 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-workload-socket\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.410426 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.410408 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-workload-certs\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.410679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.410517 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-envoy\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.410679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.410542 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-data\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.410679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.410641 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drtrv\" (UniqueName: \"kubernetes.io/projected/1411791d-2b8f-46e0-8b26-8799563d8c4b-kube-api-access-drtrv\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.410840 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.410714 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-podinfo\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.413404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.413368 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:06.515885 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.515847 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drtrv\" (UniqueName: \"kubernetes.io/projected/1411791d-2b8f-46e0-8b26-8799563d8c4b-kube-api-access-drtrv\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.516469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.516441 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-podinfo\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.516647 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.516631 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-credential-socket\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.516787 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.516774 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1411791d-2b8f-46e0-8b26-8799563d8c4b-istiod-ca-cert\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.516896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.516882 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-token\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.517003 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.516990 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-workload-socket\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.517154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.517139 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-workload-certs\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.517341 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.517328 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-envoy\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.517432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.517421 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-data\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.527117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.517949 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-data\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.529397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.529357 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1411791d-2b8f-46e0-8b26-8799563d8c4b-istiod-ca-cert\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.529397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.529364 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-workload-socket\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.544202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.536600 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-workload-certs\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.544202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.539486 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-credential-socket\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.544202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.540481 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-podinfo\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.574439 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.574394 5998 generic.go:356] "Generic (PLEG): container finished" podID="1455eaa7-24ee-45a2-bb37-155870b1ebf8" containerID="034353a3cb3c15856ad87c2ea8a22e64a142cdd60a2d9177f8e920de48a6447f" exitCode=137 Oct 30 02:13:06.574713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.574694 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-532d8a" Oct 30 02:13:06.587114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.578812 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1455eaa7-24ee-45a2-bb37-155870b1ebf8" path="/var/lib/kubelet/pods/1455eaa7-24ee-45a2-bb37-155870b1ebf8/volumes" Oct 30 02:13:06.587114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.579484 5998 scope.go:122] "RemoveContainer" containerID="034353a3cb3c15856ad87c2ea8a22e64a142cdd60a2d9177f8e920de48a6447f" Oct 30 02:13:06.587114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.586638 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drtrv\" (UniqueName: \"kubernetes.io/projected/1411791d-2b8f-46e0-8b26-8799563d8c4b-kube-api-access-drtrv\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.598127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.588644 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-token\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.625750 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.624222 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-envoy\") pod \"e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.625863 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.625833 5998 scope.go:122] "RemoveContainer" containerID="034353a3cb3c15856ad87c2ea8a22e64a142cdd60a2d9177f8e920de48a6447f" Oct 30 02:13:06.632979 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:13:06.632578 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"034353a3cb3c15856ad87c2ea8a22e64a142cdd60a2d9177f8e920de48a6447f\": container with ID starting with 034353a3cb3c15856ad87c2ea8a22e64a142cdd60a2d9177f8e920de48a6447f not found: ID does not exist" containerID="034353a3cb3c15856ad87c2ea8a22e64a142cdd60a2d9177f8e920de48a6447f" Oct 30 02:13:06.632979 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.632624 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"034353a3cb3c15856ad87c2ea8a22e64a142cdd60a2d9177f8e920de48a6447f"} err="failed to get container status \"034353a3cb3c15856ad87c2ea8a22e64a142cdd60a2d9177f8e920de48a6447f\": rpc error: code = NotFound desc = could not find container \"034353a3cb3c15856ad87c2ea8a22e64a142cdd60a2d9177f8e920de48a6447f\": container with ID starting with 034353a3cb3c15856ad87c2ea8a22e64a142cdd60a2d9177f8e920de48a6447f not found: ID does not exist" Oct 30 02:13:06.679983 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.679950 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:06.743926 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:06.743753 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:07.052380 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:07.052332 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl"] Oct 30 02:13:07.055622 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:13:07.055584 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1411791d_2b8f_46e0_8b26_8799563d8c4b.slice/crio-f45237dcab745d06976482bae8504ede03c34ccb1e9898d3602004d8d35b3203 WatchSource:0}: Error finding container f45237dcab745d06976482bae8504ede03c34ccb1e9898d3602004d8d35b3203: Status 404 returned error can't find the container with id f45237dcab745d06976482bae8504ede03c34ccb1e9898d3602004d8d35b3203 Oct 30 02:13:07.058011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:07.057975 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:13:07.058143 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:07.058060 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:13:07.058246 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:07.058197 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:13:07.389202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:07.389074 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:07.585766 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:07.585718 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" event={"ID":"1411791d-2b8f-46e0-8b26-8799563d8c4b","Type":"ContainerStarted","Data":"c0b725d9ba4c863b51985264780bce7ae018b22fcf1c74e2af9608c380d8d1da"} Oct 30 02:13:07.586306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:07.585777 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" event={"ID":"1411791d-2b8f-46e0-8b26-8799563d8c4b","Type":"ContainerStarted","Data":"f45237dcab745d06976482bae8504ede03c34ccb1e9898d3602004d8d35b3203"} Oct 30 02:13:07.589377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:07.589337 5998 generic.go:356] "Generic (PLEG): container finished" podID="9b58471f-cd08-4ba5-8c1b-89b83997dffb" containerID="b6532b6837ad200d0667c111701fe75ccbc08ea2e46f510877526c0b5771780f" exitCode=137 Oct 30 02:13:07.660827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:07.659837 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" podStartSLOduration=1.6598177440000002 podStartE2EDuration="1.659817744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:07.643600052 +0000 UTC m=+2617.222391133" watchObservedRunningTime="2026-10-30 02:13:07.659817744 +0000 UTC m=+2617.238608826" Oct 30 02:13:07.681160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:07.681067 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:07.736951 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:07.736922 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-bc3f78" Oct 30 02:13:07.867600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:07.867562 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b58471f-cd08-4ba5-8c1b-89b83997dffb-kube-api-access-58p7v\" (UniqueName: \"kubernetes.io/projected/9b58471f-cd08-4ba5-8c1b-89b83997dffb-kube-api-access-58p7v\") pod \"9b58471f-cd08-4ba5-8c1b-89b83997dffb\" (UID: \"9b58471f-cd08-4ba5-8c1b-89b83997dffb\") " Oct 30 02:13:07.869693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:07.869667 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b58471f-cd08-4ba5-8c1b-89b83997dffb-kube-api-access-58p7v" pod "9b58471f-cd08-4ba5-8c1b-89b83997dffb" (UID: "9b58471f-cd08-4ba5-8c1b-89b83997dffb"). InnerVolumeSpecName "kube-api-access-58p7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:07.969415 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:07.969380 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58p7v\" (UniqueName: \"kubernetes.io/projected/9b58471f-cd08-4ba5-8c1b-89b83997dffb-kube-api-access-58p7v\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:08.497121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.497045 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" podUID="ec8fbe57-dca0-459c-a46d-e734d53ea28a" containerName="main" containerID="cri-o://9bf3f0727bb7eda840655d244264a3ed284a6131251bef902548ba061b69bb1b" gracePeriod=45 Oct 30 02:13:08.519974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.519945 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:13:08.555194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.555149 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b58471f-cd08-4ba5-8c1b-89b83997dffb" path="/var/lib/kubelet/pods/9b58471f-cd08-4ba5-8c1b-89b83997dffb/volumes" Oct 30 02:13:08.609131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.601471 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11196-bc3f78" Oct 30 02:13:08.609131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.602339 5998 scope.go:122] "RemoveContainer" containerID="b6532b6837ad200d0667c111701fe75ccbc08ea2e46f510877526c0b5771780f" Oct 30 02:13:08.699832 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.699800 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:08.806682 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.806652 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:13:08.884319 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.884283 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec8fbe57-dca0-459c-a46d-e734d53ea28a-kube-api-access-f22wj\" (UniqueName: \"kubernetes.io/projected/ec8fbe57-dca0-459c-a46d-e734d53ea28a-kube-api-access-f22wj\") pod \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " Oct 30 02:13:08.884520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.884355 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-tmp-dir\") pod \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " Oct 30 02:13:08.884520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.884513 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec8fbe57-dca0-459c-a46d-e734d53ea28a-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8fbe57-dca0-459c-a46d-e734d53ea28a-tls-certs\") pod \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " Oct 30 02:13:08.884641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.884603 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-dshm\") pod \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " Oct 30 02:13:08.884641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.884610 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-tmp-dir" pod "ec8fbe57-dca0-459c-a46d-e734d53ea28a" (UID: "ec8fbe57-dca0-459c-a46d-e734d53ea28a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:08.884753 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.884715 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-home\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-home\") pod \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " Oct 30 02:13:08.884813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.884784 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-model-cache\") pod \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\" (UID: \"ec8fbe57-dca0-459c-a46d-e734d53ea28a\") " Oct 30 02:13:08.885010 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.884984 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-home" pod "ec8fbe57-dca0-459c-a46d-e734d53ea28a" (UID: "ec8fbe57-dca0-459c-a46d-e734d53ea28a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:08.885125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.885075 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-model-cache" pod "ec8fbe57-dca0-459c-a46d-e734d53ea28a" (UID: "ec8fbe57-dca0-459c-a46d-e734d53ea28a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:08.885459 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.885423 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:08.885561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.885463 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:08.885561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.885480 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:08.887310 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.887284 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec8fbe57-dca0-459c-a46d-e734d53ea28a-tls-certs" pod "ec8fbe57-dca0-459c-a46d-e734d53ea28a" (UID: "ec8fbe57-dca0-459c-a46d-e734d53ea28a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:13:08.887431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.887312 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8fbe57-dca0-459c-a46d-e734d53ea28a-kube-api-access-f22wj" pod "ec8fbe57-dca0-459c-a46d-e734d53ea28a" (UID: "ec8fbe57-dca0-459c-a46d-e734d53ea28a"). InnerVolumeSpecName "kube-api-access-f22wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:08.888699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.888673 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-dshm" pod "ec8fbe57-dca0-459c-a46d-e734d53ea28a" (UID: "ec8fbe57-dca0-459c-a46d-e734d53ea28a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:08.986536 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.986500 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f22wj\" (UniqueName: \"kubernetes.io/projected/ec8fbe57-dca0-459c-a46d-e734d53ea28a-kube-api-access-f22wj\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:08.986536 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.986533 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec8fbe57-dca0-459c-a46d-e734d53ea28a-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:08.986536 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:08.986542 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec8fbe57-dca0-459c-a46d-e734d53ea28a-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:09.038399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:09.038312 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:09.611803 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:09.611768 5998 generic.go:356] "Generic (PLEG): container finished" podID="ec8fbe57-dca0-459c-a46d-e734d53ea28a" containerID="9bf3f0727bb7eda840655d244264a3ed284a6131251bef902548ba061b69bb1b" exitCode=0 Oct 30 02:13:09.612342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:09.611826 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" Oct 30 02:13:09.612342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:09.611849 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" event={"ID":"ec8fbe57-dca0-459c-a46d-e734d53ea28a","Type":"ContainerDied","Data":"9bf3f0727bb7eda840655d244264a3ed284a6131251bef902548ba061b69bb1b"} Oct 30 02:13:09.612342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:09.611894 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b" event={"ID":"ec8fbe57-dca0-459c-a46d-e734d53ea28a","Type":"ContainerDied","Data":"65686d7caee6a172f1cabbcbaf80a06f0c504430132a100f3cdc30fa7907ffeb"} Oct 30 02:13:09.612342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:09.611912 5998 scope.go:122] "RemoveContainer" containerID="9bf3f0727bb7eda840655d244264a3ed284a6131251bef902548ba061b69bb1b" Oct 30 02:13:09.613364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:09.613345 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:09.614441 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:09.614421 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:13:09.628302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:09.626162 5998 scope.go:122] "RemoveContainer" containerID="9bf3f0727bb7eda840655d244264a3ed284a6131251bef902548ba061b69bb1b" Oct 30 02:13:09.631339 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:13:09.631287 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf3f0727bb7eda840655d244264a3ed284a6131251bef902548ba061b69bb1b\": container with ID starting with 9bf3f0727bb7eda840655d244264a3ed284a6131251bef902548ba061b69bb1b not found: ID does not exist" containerID="9bf3f0727bb7eda840655d244264a3ed284a6131251bef902548ba061b69bb1b" Oct 30 02:13:09.631571 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:09.631525 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf3f0727bb7eda840655d244264a3ed284a6131251bef902548ba061b69bb1b"} err="failed to get container status \"9bf3f0727bb7eda840655d244264a3ed284a6131251bef902548ba061b69bb1b\": rpc error: code = NotFound desc = could not find container \"9bf3f0727bb7eda840655d244264a3ed284a6131251bef902548ba061b69bb1b\": container with ID starting with 9bf3f0727bb7eda840655d244264a3ed284a6131251bef902548ba061b69bb1b not found: ID does not exist" Oct 30 02:13:09.703132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:09.702336 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b"] Oct 30 02:13:09.709143 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:09.708224 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-17e747/auth-test-model-17e747-kserve-6d7fc45597-s572b"] Oct 30 02:13:10.068025 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:10.067956 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:10.219508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:10.215419 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:10.541629 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:10.541591 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec8fbe57-dca0-459c-a46d-e734d53ea28a" path="/var/lib/kubelet/pods/ec8fbe57-dca0-459c-a46d-e734d53ea28a/volumes" Oct 30 02:13:10.970975 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:10.970936 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:11.808082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:11.808005 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:11.912692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:11.912653 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:12.726687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.726647 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:12.738729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.738698 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l"] Oct 30 02:13:12.739511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.739490 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ec8fbe57-dca0-459c-a46d-e734d53ea28a" containerName="main" Oct 30 02:13:12.739592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.739515 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec8fbe57-dca0-459c-a46d-e734d53ea28a" containerName="main" Oct 30 02:13:12.739723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.739710 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec8fbe57-dca0-459c-a46d-e734d53ea28a" containerName="main" Oct 30 02:13:12.745144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.745122 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l" Oct 30 02:13:12.758880 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.758856 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:13:12.771501 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.771387 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:13:12.776930 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.772499 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a405c38-2529-417f-94da-c49e43adb00b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-076fce0f-clx8l\" (UID: \"3a405c38-2529-417f-94da-c49e43adb00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l" Oct 30 02:13:12.776930 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.772583 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zpw9\" (UniqueName: \"kubernetes.io/projected/3a405c38-2529-417f-94da-c49e43adb00b-kube-api-access-7zpw9\") pod \"maas-api-revoke-sub-076fce0f-clx8l\" (UID: \"3a405c38-2529-417f-94da-c49e43adb00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l" Oct 30 02:13:12.776930 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.772690 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a405c38-2529-417f-94da-c49e43adb00b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-076fce0f-clx8l\" (UID: \"3a405c38-2529-417f-94da-c49e43adb00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l" Oct 30 02:13:12.777241 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.777117 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l"] Oct 30 02:13:12.815188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.814951 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:13:12.815188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.815132 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:13:12.874125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.874076 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a405c38-2529-417f-94da-c49e43adb00b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-076fce0f-clx8l\" (UID: \"3a405c38-2529-417f-94da-c49e43adb00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l" Oct 30 02:13:12.874305 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.874200 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a405c38-2529-417f-94da-c49e43adb00b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-076fce0f-clx8l\" (UID: \"3a405c38-2529-417f-94da-c49e43adb00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l" Oct 30 02:13:12.874305 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.874249 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zpw9\" (UniqueName: \"kubernetes.io/projected/3a405c38-2529-417f-94da-c49e43adb00b-kube-api-access-7zpw9\") pod \"maas-api-revoke-sub-076fce0f-clx8l\" (UID: \"3a405c38-2529-417f-94da-c49e43adb00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l" Oct 30 02:13:12.874818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.874792 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a405c38-2529-417f-94da-c49e43adb00b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-076fce0f-clx8l\" (UID: \"3a405c38-2529-417f-94da-c49e43adb00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l" Oct 30 02:13:12.876526 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.876507 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a405c38-2529-417f-94da-c49e43adb00b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-076fce0f-clx8l\" (UID: \"3a405c38-2529-417f-94da-c49e43adb00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l" Oct 30 02:13:12.895432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:12.895403 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zpw9\" (UniqueName: \"kubernetes.io/projected/3a405c38-2529-417f-94da-c49e43adb00b-kube-api-access-7zpw9\") pod \"maas-api-revoke-sub-076fce0f-clx8l\" (UID: \"3a405c38-2529-417f-94da-c49e43adb00b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l" Oct 30 02:13:13.059749 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:13.059668 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l" Oct 30 02:13:13.373376 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:13.373284 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l"] Oct 30 02:13:13.377104 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:13:13.377055 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a405c38_2529_417f_94da_c49e43adb00b.slice/crio-371bb76ef5df9012ccdd2c9d1f48fa123a237be734baaa2eca99fac4f0f60a93 WatchSource:0}: Error finding container 371bb76ef5df9012ccdd2c9d1f48fa123a237be734baaa2eca99fac4f0f60a93: Status 404 returned error can't find the container with id 371bb76ef5df9012ccdd2c9d1f48fa123a237be734baaa2eca99fac4f0f60a93 Oct 30 02:13:13.660438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:13.660340 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l" event={"ID":"3a405c38-2529-417f-94da-c49e43adb00b","Type":"ContainerStarted","Data":"dc64e9048efdf74cf60b5425f79299ef9d2312eb3f7e5567b307b595778531d5"} Oct 30 02:13:13.660438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:13.660408 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l" event={"ID":"3a405c38-2529-417f-94da-c49e43adb00b","Type":"ContainerStarted","Data":"371bb76ef5df9012ccdd2c9d1f48fa123a237be734baaa2eca99fac4f0f60a93"} Oct 30 02:13:13.701794 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:13.701713 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l" podStartSLOduration=1.701694845 podStartE2EDuration="1.701694845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:13.699903333 +0000 UTC m=+2623.278694414" watchObservedRunningTime="2026-10-30 02:13:13.701694845 +0000 UTC m=+2623.280485929" Oct 30 02:13:14.371702 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:13:14.371670 5998 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a405c38_2529_417f_94da_c49e43adb00b.slice/crio-dc64e9048efdf74cf60b5425f79299ef9d2312eb3f7e5567b307b595778531d5.scope\": RecentStats: unable to find data in memory cache]" Oct 30 02:13:14.535346 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:14.535310 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:14.669958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:14.669921 5998 generic.go:356] "Generic (PLEG): container finished" podID="3a405c38-2529-417f-94da-c49e43adb00b" containerID="dc64e9048efdf74cf60b5425f79299ef9d2312eb3f7e5567b307b595778531d5" exitCode=0 Oct 30 02:13:14.670184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:14.670002 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l" event={"ID":"3a405c38-2529-417f-94da-c49e43adb00b","Type":"ContainerDied","Data":"dc64e9048efdf74cf60b5425f79299ef9d2312eb3f7e5567b307b595778531d5"} Oct 30 02:13:15.836269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:15.836240 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l" Oct 30 02:13:15.910910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:15.910873 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a405c38-2529-417f-94da-c49e43adb00b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a405c38-2529-417f-94da-c49e43adb00b-maas-api-openshift-ca\") pod \"3a405c38-2529-417f-94da-c49e43adb00b\" (UID: \"3a405c38-2529-417f-94da-c49e43adb00b\") " Oct 30 02:13:15.911119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:15.911046 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a405c38-2529-417f-94da-c49e43adb00b-kube-api-access-7zpw9\" (UniqueName: \"kubernetes.io/projected/3a405c38-2529-417f-94da-c49e43adb00b-kube-api-access-7zpw9\") pod \"3a405c38-2529-417f-94da-c49e43adb00b\" (UID: \"3a405c38-2529-417f-94da-c49e43adb00b\") " Oct 30 02:13:15.911119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:15.911085 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a405c38-2529-417f-94da-c49e43adb00b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a405c38-2529-417f-94da-c49e43adb00b-maas-api-xks-ca\") pod \"3a405c38-2529-417f-94da-c49e43adb00b\" (UID: \"3a405c38-2529-417f-94da-c49e43adb00b\") " Oct 30 02:13:15.911368 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:15.911335 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a405c38-2529-417f-94da-c49e43adb00b-maas-api-openshift-ca" pod "3a405c38-2529-417f-94da-c49e43adb00b" (UID: "3a405c38-2529-417f-94da-c49e43adb00b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:15.911679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:15.911654 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a405c38-2529-417f-94da-c49e43adb00b-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:15.913417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:15.913374 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a405c38-2529-417f-94da-c49e43adb00b-maas-api-xks-ca" pod "3a405c38-2529-417f-94da-c49e43adb00b" (UID: "3a405c38-2529-417f-94da-c49e43adb00b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:13:15.913683 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:15.913659 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a405c38-2529-417f-94da-c49e43adb00b-kube-api-access-7zpw9" pod "3a405c38-2529-417f-94da-c49e43adb00b" (UID: "3a405c38-2529-417f-94da-c49e43adb00b"). InnerVolumeSpecName "kube-api-access-7zpw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:16.007228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:16.007118 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:16.012818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:16.012790 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zpw9\" (UniqueName: \"kubernetes.io/projected/3a405c38-2529-417f-94da-c49e43adb00b-kube-api-access-7zpw9\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.012970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:16.012823 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a405c38-2529-417f-94da-c49e43adb00b-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.168239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:16.168190 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb"] Oct 30 02:13:16.210154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:16.208659 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:16.685700 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:16.685668 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l" Oct 30 02:13:16.685887 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:16.685700 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l" event={"ID":"3a405c38-2529-417f-94da-c49e43adb00b","Type":"ContainerDied","Data":"371bb76ef5df9012ccdd2c9d1f48fa123a237be734baaa2eca99fac4f0f60a93"} Oct 30 02:13:16.685887 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:16.685750 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="371bb76ef5df9012ccdd2c9d1f48fa123a237be734baaa2eca99fac4f0f60a93" Oct 30 02:13:17.694251 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:17.694219 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:19.188637 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:19.188604 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:20.849534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:20.849501 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:20.972208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:20.972156 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:21.130068 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:21.129981 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:22.375771 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:22.375731 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:22.604611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:22.604570 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:23.106306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.106271 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb"] Oct 30 02:13:23.106824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.106809 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3a405c38-2529-417f-94da-c49e43adb00b" containerName="revoke-keys" Oct 30 02:13:23.106923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.106825 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a405c38-2529-417f-94da-c49e43adb00b" containerName="revoke-keys" Oct 30 02:13:23.106982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.106928 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a405c38-2529-417f-94da-c49e43adb00b" containerName="revoke-keys" Oct 30 02:13:23.110829 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.110800 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb" Oct 30 02:13:23.123950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.123922 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:13:23.123950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.123909 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:13:23.134240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.134211 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:13:23.134397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.134360 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:13:23.138577 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.138551 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb"] Oct 30 02:13:23.217145 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.217107 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c2b7588-719c-4b04-9a95-963da9214886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d5195dc-qrgmb\" (UID: \"7c2b7588-719c-4b04-9a95-963da9214886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb" Oct 30 02:13:23.217342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.217239 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c2b7588-719c-4b04-9a95-963da9214886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d5195dc-qrgmb\" (UID: \"7c2b7588-719c-4b04-9a95-963da9214886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb" Oct 30 02:13:23.217418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.217342 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk2zw\" (UniqueName: \"kubernetes.io/projected/7c2b7588-719c-4b04-9a95-963da9214886-kube-api-access-hk2zw\") pod \"maas-api-revoke-sub-2d5195dc-qrgmb\" (UID: \"7c2b7588-719c-4b04-9a95-963da9214886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb" Oct 30 02:13:23.319146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.319110 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c2b7588-719c-4b04-9a95-963da9214886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d5195dc-qrgmb\" (UID: \"7c2b7588-719c-4b04-9a95-963da9214886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb" Oct 30 02:13:23.319351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.319197 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c2b7588-719c-4b04-9a95-963da9214886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d5195dc-qrgmb\" (UID: \"7c2b7588-719c-4b04-9a95-963da9214886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb" Oct 30 02:13:23.319351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.319247 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk2zw\" (UniqueName: \"kubernetes.io/projected/7c2b7588-719c-4b04-9a95-963da9214886-kube-api-access-hk2zw\") pod \"maas-api-revoke-sub-2d5195dc-qrgmb\" (UID: \"7c2b7588-719c-4b04-9a95-963da9214886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb" Oct 30 02:13:23.319816 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.319788 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c2b7588-719c-4b04-9a95-963da9214886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d5195dc-qrgmb\" (UID: \"7c2b7588-719c-4b04-9a95-963da9214886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb" Oct 30 02:13:23.321461 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.321443 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c2b7588-719c-4b04-9a95-963da9214886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d5195dc-qrgmb\" (UID: \"7c2b7588-719c-4b04-9a95-963da9214886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb" Oct 30 02:13:23.340458 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.340433 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk2zw\" (UniqueName: \"kubernetes.io/projected/7c2b7588-719c-4b04-9a95-963da9214886-kube-api-access-hk2zw\") pod \"maas-api-revoke-sub-2d5195dc-qrgmb\" (UID: \"7c2b7588-719c-4b04-9a95-963da9214886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb" Oct 30 02:13:23.426417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.426384 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb" Oct 30 02:13:23.735312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.735209 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb"] Oct 30 02:13:23.739072 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:13:23.739040 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c2b7588_719c_4b04_9a95_963da9214886.slice/crio-a24bf4d17e919a21655c1e245c6ea053a1df3791845f89f317edf35720bff263 WatchSource:0}: Error finding container a24bf4d17e919a21655c1e245c6ea053a1df3791845f89f317edf35720bff263: Status 404 returned error can't find the container with id a24bf4d17e919a21655c1e245c6ea053a1df3791845f89f317edf35720bff263 Oct 30 02:13:23.858972 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:23.858930 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:24.283165 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.282205 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg"] Oct 30 02:13:24.296139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.286630 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.302750 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.302712 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:24.309936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.309906 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a0fc73-openshift-default-dockercfg-gn4z5\"" Oct 30 02:13:24.316254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.316070 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg"] Oct 30 02:13:24.409394 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.409357 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:24.438982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.438948 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-token\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.439481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.439049 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57be1a46-2149-46f4-a048-b099d67d3a6b-istiod-ca-cert\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.439481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.439149 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-data\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.439481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.439177 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-podinfo\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.439481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.439213 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzmcf\" (UniqueName: \"kubernetes.io/projected/57be1a46-2149-46f4-a048-b099d67d3a6b-kube-api-access-vzmcf\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.439481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.439247 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-workload-certs\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.439481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.439363 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-envoy\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.439481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.439423 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-workload-socket\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.439481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.439482 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-credential-socket\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.529171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.518586 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:24.542105 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.541985 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-token\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.542105 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.542079 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57be1a46-2149-46f4-a048-b099d67d3a6b-istiod-ca-cert\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.542354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.542156 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-data\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.542354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.542183 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-podinfo\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.542354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.542230 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzmcf\" (UniqueName: \"kubernetes.io/projected/57be1a46-2149-46f4-a048-b099d67d3a6b-kube-api-access-vzmcf\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.542354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.542265 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-workload-certs\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.542354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.542318 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-envoy\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.542720 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.542363 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-workload-socket\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.542720 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.542418 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-credential-socket\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.550123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.542883 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-credential-socket\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.550123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.543808 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57be1a46-2149-46f4-a048-b099d67d3a6b-istiod-ca-cert\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.550123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.544109 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-data\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.550123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.545338 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-workload-certs\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.550123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.546116 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-workload-socket\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.550123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.548518 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-envoy\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.570809 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.570773 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-podinfo\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.580126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.574715 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzmcf\" (UniqueName: \"kubernetes.io/projected/57be1a46-2149-46f4-a048-b099d67d3a6b-kube-api-access-vzmcf\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.582666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.582635 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-token\") pod \"e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.610815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.607354 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:24.759888 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.759857 5998 generic.go:356] "Generic (PLEG): container finished" podID="7c2b7588-719c-4b04-9a95-963da9214886" containerID="7e46698ecad9bee81bff804756d76cdd723e9a95be93d6b79264fe6c16f8bdac" exitCode=0 Oct 30 02:13:24.760071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.759948 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb" event={"ID":"7c2b7588-719c-4b04-9a95-963da9214886","Type":"ContainerDied","Data":"7e46698ecad9bee81bff804756d76cdd723e9a95be93d6b79264fe6c16f8bdac"} Oct 30 02:13:24.760071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.760014 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb" event={"ID":"7c2b7588-719c-4b04-9a95-963da9214886","Type":"ContainerStarted","Data":"a24bf4d17e919a21655c1e245c6ea053a1df3791845f89f317edf35720bff263"} Oct 30 02:13:24.886125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.886014 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb"] Oct 30 02:13:24.886326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.886260 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" podUID="9bcd8c28-6f20-4e7e-b682-729266e83210" containerName="istio-proxy" containerID="cri-o://85a9fb6e309aa69dca955b8afc41a231c347ad7046cf26362ff64fa031f96e6a" gracePeriod=30 Oct 30 02:13:24.983380 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.983352 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg"] Oct 30 02:13:24.986505 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:13:24.986461 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57be1a46_2149_46f4_a048_b099d67d3a6b.slice/crio-67a794c4f3766f4cca1a020412dcd45c1b5e1c6a3481f62c64c8a4f3c2098cad WatchSource:0}: Error finding container 67a794c4f3766f4cca1a020412dcd45c1b5e1c6a3481f62c64c8a4f3c2098cad: Status 404 returned error can't find the container with id 67a794c4f3766f4cca1a020412dcd45c1b5e1c6a3481f62c64c8a4f3c2098cad Oct 30 02:13:24.989217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.989181 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:13:24.989313 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.989261 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:13:24.989313 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:24.989306 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:13:25.228983 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:13:25.228942 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:13:25.770461 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:25.770422 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" event={"ID":"57be1a46-2149-46f4-a048-b099d67d3a6b","Type":"ContainerStarted","Data":"122a350bf9d6bd79a9e50c83b560c5ddf2991e2b97e66d4f77477449c28424b6"} Oct 30 02:13:25.771334 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:25.771307 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" event={"ID":"57be1a46-2149-46f4-a048-b099d67d3a6b","Type":"ContainerStarted","Data":"67a794c4f3766f4cca1a020412dcd45c1b5e1c6a3481f62c64c8a4f3c2098cad"} Oct 30 02:13:25.831605 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:25.831552 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" podStartSLOduration=1.83153685 podStartE2EDuration="1.83153685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:25.831166406 +0000 UTC m=+2635.409957489" watchObservedRunningTime="2026-10-30 02:13:25.83153685 +0000 UTC m=+2635.410327930" Oct 30 02:13:25.883636 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:25.883598 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:25.981013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:25.980969 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb" Oct 30 02:13:26.077330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:26.077226 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c2b7588-719c-4b04-9a95-963da9214886-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c2b7588-719c-4b04-9a95-963da9214886-maas-api-openshift-ca\") pod \"7c2b7588-719c-4b04-9a95-963da9214886\" (UID: \"7c2b7588-719c-4b04-9a95-963da9214886\") " Oct 30 02:13:26.077330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:26.077280 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c2b7588-719c-4b04-9a95-963da9214886-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c2b7588-719c-4b04-9a95-963da9214886-maas-api-xks-ca\") pod \"7c2b7588-719c-4b04-9a95-963da9214886\" (UID: \"7c2b7588-719c-4b04-9a95-963da9214886\") " Oct 30 02:13:26.077582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:26.077421 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c2b7588-719c-4b04-9a95-963da9214886-kube-api-access-hk2zw\" (UniqueName: \"kubernetes.io/projected/7c2b7588-719c-4b04-9a95-963da9214886-kube-api-access-hk2zw\") pod \"7c2b7588-719c-4b04-9a95-963da9214886\" (UID: \"7c2b7588-719c-4b04-9a95-963da9214886\") " Oct 30 02:13:26.077702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:26.077673 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c2b7588-719c-4b04-9a95-963da9214886-maas-api-openshift-ca" pod "7c2b7588-719c-4b04-9a95-963da9214886" (UID: "7c2b7588-719c-4b04-9a95-963da9214886"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:26.078114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:26.078072 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c2b7588-719c-4b04-9a95-963da9214886-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:26.079727 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:26.079694 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c2b7588-719c-4b04-9a95-963da9214886-kube-api-access-hk2zw" pod "7c2b7588-719c-4b04-9a95-963da9214886" (UID: "7c2b7588-719c-4b04-9a95-963da9214886"). InnerVolumeSpecName "kube-api-access-hk2zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:26.079902 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:26.079883 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c2b7588-719c-4b04-9a95-963da9214886-maas-api-xks-ca" pod "7c2b7588-719c-4b04-9a95-963da9214886" (UID: "7c2b7588-719c-4b04-9a95-963da9214886"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:13:26.179668 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:26.179629 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c2b7588-719c-4b04-9a95-963da9214886-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:26.179668 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:26.179662 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk2zw\" (UniqueName: \"kubernetes.io/projected/7c2b7588-719c-4b04-9a95-963da9214886-kube-api-access-hk2zw\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:26.612651 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:26.612612 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:26.618580 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:26.618552 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:26.779474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:26.779431 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb" Oct 30 02:13:26.779474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:26.779443 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb" event={"ID":"7c2b7588-719c-4b04-9a95-963da9214886","Type":"ContainerDied","Data":"a24bf4d17e919a21655c1e245c6ea053a1df3791845f89f317edf35720bff263"} Oct 30 02:13:26.780010 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:26.779504 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a24bf4d17e919a21655c1e245c6ea053a1df3791845f89f317edf35720bff263" Oct 30 02:13:26.780196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:26.780172 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:26.780798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:26.780781 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:13:27.616647 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:27.616611 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:28.171544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:28.171510 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:28.284842 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:28.284809 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:29.449144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:29.449107 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:29.625971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:29.625938 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn"] Oct 30 02:13:29.626187 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:29.626158 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" podUID="2b27c952-8035-4847-b243-4471250d3d09" containerName="istio-proxy" containerID="cri-o://ba8d5fa15cf551963cebbe78d9521409b26dff8203ffc421c1da7595761dff35" gracePeriod=30 Oct 30 02:13:30.011878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.011851 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:30.124543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.122430 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:30.170498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.170396 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 02:13:30.236487 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.236452 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-podinfo\") pod \"9bcd8c28-6f20-4e7e-b682-729266e83210\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " Oct 30 02:13:30.236902 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.236879 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-credential-socket\") pod \"9bcd8c28-6f20-4e7e-b682-729266e83210\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " Oct 30 02:13:30.237596 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.237491 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-credential-socket" pod "9bcd8c28-6f20-4e7e-b682-729266e83210" (UID: "9bcd8c28-6f20-4e7e-b682-729266e83210"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:30.239824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.238272 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-envoy\") pod \"9bcd8c28-6f20-4e7e-b682-729266e83210\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " Oct 30 02:13:30.240209 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.240189 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-workload-certs\") pod \"9bcd8c28-6f20-4e7e-b682-729266e83210\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " Oct 30 02:13:30.240670 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.240643 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-data\") pod \"9bcd8c28-6f20-4e7e-b682-729266e83210\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " Oct 30 02:13:30.241085 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.240586 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-workload-certs" pod "9bcd8c28-6f20-4e7e-b682-729266e83210" (UID: "9bcd8c28-6f20-4e7e-b682-729266e83210"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:30.241242 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.240973 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-data" pod "9bcd8c28-6f20-4e7e-b682-729266e83210" (UID: "9bcd8c28-6f20-4e7e-b682-729266e83210"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:30.241364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.241349 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bcd8c28-6f20-4e7e-b682-729266e83210-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcd8c28-6f20-4e7e-b682-729266e83210-istiod-ca-cert\") pod \"9bcd8c28-6f20-4e7e-b682-729266e83210\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " Oct 30 02:13:30.241861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.241747 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bcd8c28-6f20-4e7e-b682-729266e83210-istiod-ca-cert" pod "9bcd8c28-6f20-4e7e-b682-729266e83210" (UID: "9bcd8c28-6f20-4e7e-b682-729266e83210"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:30.243435 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.241815 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-token\" (UniqueName: \"kubernetes.io/projected/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-token\") pod \"9bcd8c28-6f20-4e7e-b682-729266e83210\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " Oct 30 02:13:30.243670 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.243652 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bcd8c28-6f20-4e7e-b682-729266e83210-kube-api-access-gx6wh\" (UniqueName: \"kubernetes.io/projected/9bcd8c28-6f20-4e7e-b682-729266e83210-kube-api-access-gx6wh\") pod \"9bcd8c28-6f20-4e7e-b682-729266e83210\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " Oct 30 02:13:30.244406 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.244385 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-workload-socket\") pod \"9bcd8c28-6f20-4e7e-b682-729266e83210\" (UID: \"9bcd8c28-6f20-4e7e-b682-729266e83210\") " Oct 30 02:13:30.275792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.270775 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-podinfo" pod "9bcd8c28-6f20-4e7e-b682-729266e83210" (UID: "9bcd8c28-6f20-4e7e-b682-729266e83210"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:13:30.276662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.276631 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:30.276662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.276662 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:30.276903 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.276677 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bcd8c28-6f20-4e7e-b682-729266e83210-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:30.276903 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.276689 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:30.276903 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.276701 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:30.277225 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.277199 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-workload-socket" pod "9bcd8c28-6f20-4e7e-b682-729266e83210" (UID: "9bcd8c28-6f20-4e7e-b682-729266e83210"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:30.390404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.378297 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-token" pod "9bcd8c28-6f20-4e7e-b682-729266e83210" (UID: "9bcd8c28-6f20-4e7e-b682-729266e83210"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:30.390404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.379465 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:30.390404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.379494 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:30.390404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.385328 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-envoy" pod "9bcd8c28-6f20-4e7e-b682-729266e83210" (UID: "9bcd8c28-6f20-4e7e-b682-729266e83210"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:30.390404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.390308 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bcd8c28-6f20-4e7e-b682-729266e83210-kube-api-access-gx6wh" pod "9bcd8c28-6f20-4e7e-b682-729266e83210" (UID: "9bcd8c28-6f20-4e7e-b682-729266e83210"). InnerVolumeSpecName "kube-api-access-gx6wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:30.481314 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.481231 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx6wh\" (UniqueName: \"kubernetes.io/projected/9bcd8c28-6f20-4e7e-b682-729266e83210-kube-api-access-gx6wh\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:30.481764 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.481739 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bcd8c28-6f20-4e7e-b682-729266e83210-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:30.814359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.814254 5998 generic.go:356] "Generic (PLEG): container finished" podID="9bcd8c28-6f20-4e7e-b682-729266e83210" containerID="85a9fb6e309aa69dca955b8afc41a231c347ad7046cf26362ff64fa031f96e6a" exitCode=0 Oct 30 02:13:30.814359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.814328 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" Oct 30 02:13:30.814359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.814342 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" event={"ID":"9bcd8c28-6f20-4e7e-b682-729266e83210","Type":"ContainerDied","Data":"85a9fb6e309aa69dca955b8afc41a231c347ad7046cf26362ff64fa031f96e6a"} Oct 30 02:13:30.814662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.814434 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb" event={"ID":"9bcd8c28-6f20-4e7e-b682-729266e83210","Type":"ContainerDied","Data":"10ec2a7262e78f9fd1e70caee7975f936f6458ac4325fbbee78de810b61d7459"} Oct 30 02:13:30.814662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.814471 5998 scope.go:122] "RemoveContainer" containerID="85a9fb6e309aa69dca955b8afc41a231c347ad7046cf26362ff64fa031f96e6a" Oct 30 02:13:30.826043 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.826021 5998 scope.go:122] "RemoveContainer" containerID="85a9fb6e309aa69dca955b8afc41a231c347ad7046cf26362ff64fa031f96e6a" Oct 30 02:13:30.826394 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:13:30.826370 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a9fb6e309aa69dca955b8afc41a231c347ad7046cf26362ff64fa031f96e6a\": container with ID starting with 85a9fb6e309aa69dca955b8afc41a231c347ad7046cf26362ff64fa031f96e6a not found: ID does not exist" containerID="85a9fb6e309aa69dca955b8afc41a231c347ad7046cf26362ff64fa031f96e6a" Oct 30 02:13:30.826506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.826404 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a9fb6e309aa69dca955b8afc41a231c347ad7046cf26362ff64fa031f96e6a"} err="failed to get container status \"85a9fb6e309aa69dca955b8afc41a231c347ad7046cf26362ff64fa031f96e6a\": rpc error: code = NotFound desc = could not find container \"85a9fb6e309aa69dca955b8afc41a231c347ad7046cf26362ff64fa031f96e6a\": container with ID starting with 85a9fb6e309aa69dca955b8afc41a231c347ad7046cf26362ff64fa031f96e6a not found: ID does not exist" Oct 30 02:13:30.863356 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.863320 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb"] Oct 30 02:13:30.873389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:30.873357 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7l9sb"] Oct 30 02:13:31.211381 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.211341 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" podUID="a2cd2f87-e701-4574-8193-901a704f2c80" containerName="main" containerID="cri-o://23cc555a8effafe828290b18b4c40d79d05d087abb664cb8ba6b6b4ee19ea542" gracePeriod=45 Oct 30 02:13:31.230374 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.230308 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:31.501721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.501693 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:13:31.599098 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.599059 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-tmp-dir\") pod \"a2cd2f87-e701-4574-8193-901a704f2c80\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " Oct 30 02:13:31.599263 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.599141 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2cd2f87-e701-4574-8193-901a704f2c80-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2cd2f87-e701-4574-8193-901a704f2c80-tls-certs\") pod \"a2cd2f87-e701-4574-8193-901a704f2c80\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " Oct 30 02:13:31.599263 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.599244 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2cd2f87-e701-4574-8193-901a704f2c80-kube-api-access-7rjvl\" (UniqueName: \"kubernetes.io/projected/a2cd2f87-e701-4574-8193-901a704f2c80-kube-api-access-7rjvl\") pod \"a2cd2f87-e701-4574-8193-901a704f2c80\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " Oct 30 02:13:31.599354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.599329 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-dshm\") pod \"a2cd2f87-e701-4574-8193-901a704f2c80\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " Oct 30 02:13:31.599354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.599332 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-tmp-dir" pod "a2cd2f87-e701-4574-8193-901a704f2c80" (UID: "a2cd2f87-e701-4574-8193-901a704f2c80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:31.599442 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.599369 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-home\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-home\") pod \"a2cd2f87-e701-4574-8193-901a704f2c80\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " Oct 30 02:13:31.599497 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.599469 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-model-cache\") pod \"a2cd2f87-e701-4574-8193-901a704f2c80\" (UID: \"a2cd2f87-e701-4574-8193-901a704f2c80\") " Oct 30 02:13:31.599721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.599693 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-home" pod "a2cd2f87-e701-4574-8193-901a704f2c80" (UID: "a2cd2f87-e701-4574-8193-901a704f2c80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:31.599883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.599776 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-model-cache" pod "a2cd2f87-e701-4574-8193-901a704f2c80" (UID: "a2cd2f87-e701-4574-8193-901a704f2c80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:31.600108 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.600061 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:31.600189 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.600109 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:31.600189 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.600124 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:31.601772 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.601697 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2cd2f87-e701-4574-8193-901a704f2c80-kube-api-access-7rjvl" pod "a2cd2f87-e701-4574-8193-901a704f2c80" (UID: "a2cd2f87-e701-4574-8193-901a704f2c80"). InnerVolumeSpecName "kube-api-access-7rjvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:31.601772 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.601727 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2cd2f87-e701-4574-8193-901a704f2c80-tls-certs" pod "a2cd2f87-e701-4574-8193-901a704f2c80" (UID: "a2cd2f87-e701-4574-8193-901a704f2c80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:13:31.601965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.601776 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-dshm" pod "a2cd2f87-e701-4574-8193-901a704f2c80" (UID: "a2cd2f87-e701-4574-8193-901a704f2c80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:31.701983 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.701942 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2cd2f87-e701-4574-8193-901a704f2c80-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:31.701983 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.701978 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rjvl\" (UniqueName: \"kubernetes.io/projected/a2cd2f87-e701-4574-8193-901a704f2c80-kube-api-access-7rjvl\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:31.702268 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.701993 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2cd2f87-e701-4574-8193-901a704f2c80-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:31.826760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.826671 5998 generic.go:356] "Generic (PLEG): container finished" podID="a2cd2f87-e701-4574-8193-901a704f2c80" containerID="23cc555a8effafe828290b18b4c40d79d05d087abb664cb8ba6b6b4ee19ea542" exitCode=0 Oct 30 02:13:31.826760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.826722 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" Oct 30 02:13:31.826959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.826725 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" event={"ID":"a2cd2f87-e701-4574-8193-901a704f2c80","Type":"ContainerDied","Data":"23cc555a8effafe828290b18b4c40d79d05d087abb664cb8ba6b6b4ee19ea542"} Oct 30 02:13:31.826959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.826837 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb" event={"ID":"a2cd2f87-e701-4574-8193-901a704f2c80","Type":"ContainerDied","Data":"a1de1b136ecb18fa6a9cdb262381798f29f9344da065732cfeef1fd54c9506ed"} Oct 30 02:13:31.826959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.826860 5998 scope.go:122] "RemoveContainer" containerID="23cc555a8effafe828290b18b4c40d79d05d087abb664cb8ba6b6b4ee19ea542" Oct 30 02:13:31.838783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.838761 5998 scope.go:122] "RemoveContainer" containerID="23cc555a8effafe828290b18b4c40d79d05d087abb664cb8ba6b6b4ee19ea542" Oct 30 02:13:31.839119 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:13:31.839080 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23cc555a8effafe828290b18b4c40d79d05d087abb664cb8ba6b6b4ee19ea542\": container with ID starting with 23cc555a8effafe828290b18b4c40d79d05d087abb664cb8ba6b6b4ee19ea542 not found: ID does not exist" containerID="23cc555a8effafe828290b18b4c40d79d05d087abb664cb8ba6b6b4ee19ea542" Oct 30 02:13:31.839215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.839132 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23cc555a8effafe828290b18b4c40d79d05d087abb664cb8ba6b6b4ee19ea542"} err="failed to get container status \"23cc555a8effafe828290b18b4c40d79d05d087abb664cb8ba6b6b4ee19ea542\": rpc error: code = NotFound desc = could not find container \"23cc555a8effafe828290b18b4c40d79d05d087abb664cb8ba6b6b4ee19ea542\": container with ID starting with 23cc555a8effafe828290b18b4c40d79d05d087abb664cb8ba6b6b4ee19ea542 not found: ID does not exist" Oct 30 02:13:31.879931 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.879894 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb"] Oct 30 02:13:31.889715 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:31.889686 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2819d2/auth-test-model-2819d2-kserve-5fc754cb9c-nlclb"] Oct 30 02:13:32.538888 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:32.538852 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bcd8c28-6f20-4e7e-b682-729266e83210" path="/var/lib/kubelet/pods/9bcd8c28-6f20-4e7e-b682-729266e83210/volumes" Oct 30 02:13:32.539548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:32.539523 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2cd2f87-e701-4574-8193-901a704f2c80" path="/var/lib/kubelet/pods/a2cd2f87-e701-4574-8193-901a704f2c80/volumes" Oct 30 02:13:32.594440 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:32.594407 5998 scope.go:122] "RemoveContainer" containerID="5e22f93ed625aac5270ad6fe4511877b4d3078238f8c4643c554a06de95bf35f" Oct 30 02:13:32.610987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:32.610958 5998 scope.go:122] "RemoveContainer" containerID="d2303845278c70cf593f93ec37ff12959676a62555f1e117f0fd352245652911" Oct 30 02:13:32.622630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:32.622603 5998 scope.go:122] "RemoveContainer" containerID="366db7a9733d7971bba78b9b21bcd37984922e3833bdb0a89f2e2c4e40b524dd" Oct 30 02:13:32.632151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:32.632127 5998 scope.go:122] "RemoveContainer" containerID="84a4739e0da7e435d277c9a0d7732550d1459a8c3f73203cee85dec1cb873174" Oct 30 02:13:32.642268 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:32.642242 5998 scope.go:122] "RemoveContainer" containerID="d3e81082d2b84227187b2540ce57626a3304726658e1b8d73a4d49e9b1e03e51" Oct 30 02:13:32.651134 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:32.651110 5998 scope.go:122] "RemoveContainer" containerID="158aa69ded596f4e63ca223331161a0ebe4c338b4b6359dad700fefa77918526" Oct 30 02:13:32.660293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:32.660271 5998 scope.go:122] "RemoveContainer" containerID="d7841df9c853741049a9a34f4c31d59dbcf8d01b191ebae3fed29a090371731b" Oct 30 02:13:32.788269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:32.788239 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:34.425277 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:34.425247 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:34.547494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:34.547422 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:34.657915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:34.657869 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:34.856727 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:34.856694 5998 generic.go:356] "Generic (PLEG): container finished" podID="2b27c952-8035-4847-b243-4471250d3d09" containerID="ba8d5fa15cf551963cebbe78d9521409b26dff8203ffc421c1da7595761dff35" exitCode=0 Oct 30 02:13:34.856882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:34.856846 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" event={"ID":"2b27c952-8035-4847-b243-4471250d3d09","Type":"ContainerDied","Data":"ba8d5fa15cf551963cebbe78d9521409b26dff8203ffc421c1da7595761dff35"} Oct 30 02:13:34.898866 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:34.898841 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:13:35.033581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.033490 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm"] Oct 30 02:13:35.038311 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.038283 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e15a0e69-s98dm"] Oct 30 02:13:35.058409 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.058382 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b27c952-8035-4847-b243-4471250d3d09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b27c952-8035-4847-b243-4471250d3d09-istio-podinfo\") pod \"2b27c952-8035-4847-b243-4471250d3d09\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " Oct 30 02:13:35.058590 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.058450 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b27c952-8035-4847-b243-4471250d3d09-kube-api-access-xn76d\" (UniqueName: \"kubernetes.io/projected/2b27c952-8035-4847-b243-4471250d3d09-kube-api-access-xn76d\") pod \"2b27c952-8035-4847-b243-4471250d3d09\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " Oct 30 02:13:35.058590 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.058491 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-workload-socket\") pod \"2b27c952-8035-4847-b243-4471250d3d09\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " Oct 30 02:13:35.058710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.058609 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-istio-data\") pod \"2b27c952-8035-4847-b243-4471250d3d09\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " Oct 30 02:13:35.058710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.058661 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-istio-envoy\") pod \"2b27c952-8035-4847-b243-4471250d3d09\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " Oct 30 02:13:35.058836 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.058712 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b27c952-8035-4847-b243-4471250d3d09-istio-token\" (UniqueName: \"kubernetes.io/projected/2b27c952-8035-4847-b243-4471250d3d09-istio-token\") pod \"2b27c952-8035-4847-b243-4471250d3d09\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " Oct 30 02:13:35.058836 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.058745 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b27c952-8035-4847-b243-4471250d3d09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b27c952-8035-4847-b243-4471250d3d09-istiod-ca-cert\") pod \"2b27c952-8035-4847-b243-4471250d3d09\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " Oct 30 02:13:35.058950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.058836 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-credential-socket\") pod \"2b27c952-8035-4847-b243-4471250d3d09\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " Oct 30 02:13:35.058950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.058903 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-workload-certs\") pod \"2b27c952-8035-4847-b243-4471250d3d09\" (UID: \"2b27c952-8035-4847-b243-4471250d3d09\") " Oct 30 02:13:35.061450 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.061413 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-workload-certs" pod "2b27c952-8035-4847-b243-4471250d3d09" (UID: "2b27c952-8035-4847-b243-4471250d3d09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:35.061572 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.061502 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-istio-data" pod "2b27c952-8035-4847-b243-4471250d3d09" (UID: "2b27c952-8035-4847-b243-4471250d3d09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:35.062342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.062315 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-credential-socket" pod "2b27c952-8035-4847-b243-4471250d3d09" (UID: "2b27c952-8035-4847-b243-4471250d3d09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:35.062481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.062455 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-workload-socket" pod "2b27c952-8035-4847-b243-4471250d3d09" (UID: "2b27c952-8035-4847-b243-4471250d3d09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:35.065160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.062719 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-istio-envoy" pod "2b27c952-8035-4847-b243-4471250d3d09" (UID: "2b27c952-8035-4847-b243-4471250d3d09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:35.065160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.062737 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b27c952-8035-4847-b243-4471250d3d09-kube-api-access-xn76d" pod "2b27c952-8035-4847-b243-4471250d3d09" (UID: "2b27c952-8035-4847-b243-4471250d3d09"). InnerVolumeSpecName "kube-api-access-xn76d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:35.065160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.062794 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b27c952-8035-4847-b243-4471250d3d09-istio-podinfo" pod "2b27c952-8035-4847-b243-4471250d3d09" (UID: "2b27c952-8035-4847-b243-4471250d3d09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:13:35.065160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.062866 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b27c952-8035-4847-b243-4471250d3d09-istiod-ca-cert" pod "2b27c952-8035-4847-b243-4471250d3d09" (UID: "2b27c952-8035-4847-b243-4471250d3d09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:35.065160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.064235 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b27c952-8035-4847-b243-4471250d3d09-istio-token" pod "2b27c952-8035-4847-b243-4471250d3d09" (UID: "2b27c952-8035-4847-b243-4471250d3d09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:35.160379 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.160336 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn76d\" (UniqueName: \"kubernetes.io/projected/2b27c952-8035-4847-b243-4471250d3d09-kube-api-access-xn76d\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:35.160379 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.160368 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:35.160379 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.160378 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:35.160379 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.160387 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:35.160744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.160395 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b27c952-8035-4847-b243-4471250d3d09-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:35.160744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.160403 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b27c952-8035-4847-b243-4471250d3d09-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:35.160744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.160410 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:35.160744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.160418 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b27c952-8035-4847-b243-4471250d3d09-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:35.160744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.160426 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b27c952-8035-4847-b243-4471250d3d09-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:35.864982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.864958 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" Oct 30 02:13:35.864982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.864967 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn" event={"ID":"2b27c952-8035-4847-b243-4471250d3d09","Type":"ContainerDied","Data":"d64dd504704dc4fa4a09b68354fa322f86c309aadbcc672b1a842a3e0d5a8b86"} Oct 30 02:13:35.865545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.865036 5998 scope.go:122] "RemoveContainer" containerID="ba8d5fa15cf551963cebbe78d9521409b26dff8203ffc421c1da7595761dff35" Oct 30 02:13:35.926868 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.926834 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn"] Oct 30 02:13:35.936076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:35.936048 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-wwmvn"] Oct 30 02:13:36.067713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:36.067681 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:36.539265 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:36.539228 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b27c952-8035-4847-b243-4471250d3d09" path="/var/lib/kubelet/pods/2b27c952-8035-4847-b243-4471250d3d09/volumes" Oct 30 02:13:36.539770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:36.539754 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a240736-76fb-44ae-a56a-f99f5e84b37c" path="/var/lib/kubelet/pods/4a240736-76fb-44ae-a56a-f99f5e84b37c/volumes" Oct 30 02:13:37.304017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:37.303973 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:37.798507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:37.798473 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:39.581585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:39.581549 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:41.522067 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:41.522035 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:43.023398 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:43.023341 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:43.274359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:43.274274 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:44.254115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:44.248871 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np"] Oct 30 02:13:44.254115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:44.251303 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:45.017804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:45.017764 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:46.633013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:46.632974 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:48.232661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:48.232622 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:49.032012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:49.031974 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l"] Oct 30 02:13:49.042069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:49.042040 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56ff3412-ngq2l"] Oct 30 02:13:49.986269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:49.986230 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:50.396870 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:50.396778 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl"] Oct 30 02:13:50.420410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:50.420374 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:50.570293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:50.570251 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0bfdf1b-a614-4293-911d-f2c8ce5be38e" path="/var/lib/kubelet/pods/f0bfdf1b-a614-4293-911d-f2c8ce5be38e/volumes" Oct 30 02:13:51.731829 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:51.731795 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:51.842696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:51.842662 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:51.951599 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:51.951564 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:52.229118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.229068 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj"] Oct 30 02:13:52.230475 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.230435 5998 cpu_manager.go:414] "containerMap: removing container" podUID="9bcd8c28-6f20-4e7e-b682-729266e83210" containerName="istio-proxy" Oct 30 02:13:52.230823 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.230806 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bcd8c28-6f20-4e7e-b682-729266e83210" containerName="istio-proxy" Oct 30 02:13:52.230968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.230956 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7c2b7588-719c-4b04-9a95-963da9214886" containerName="revoke-keys" Oct 30 02:13:52.231072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.231050 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c2b7588-719c-4b04-9a95-963da9214886" containerName="revoke-keys" Oct 30 02:13:52.231237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.231114 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a2cd2f87-e701-4574-8193-901a704f2c80" containerName="main" Oct 30 02:13:52.231237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.231125 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2cd2f87-e701-4574-8193-901a704f2c80" containerName="main" Oct 30 02:13:52.231237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.231144 5998 cpu_manager.go:414] "containerMap: removing container" podUID="2b27c952-8035-4847-b243-4471250d3d09" containerName="istio-proxy" Oct 30 02:13:52.231237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.231153 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b27c952-8035-4847-b243-4471250d3d09" containerName="istio-proxy" Oct 30 02:13:52.231372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.231341 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2cd2f87-e701-4574-8193-901a704f2c80" containerName="main" Oct 30 02:13:52.231372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.231358 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b27c952-8035-4847-b243-4471250d3d09" containerName="istio-proxy" Oct 30 02:13:52.231481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.231378 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bcd8c28-6f20-4e7e-b682-729266e83210" containerName="istio-proxy" Oct 30 02:13:52.231481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.231389 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c2b7588-719c-4b04-9a95-963da9214886" containerName="revoke-keys" Oct 30 02:13:52.237183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.237158 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.251307 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.251282 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-6b8bfa-openshift-default-dockercfg-gc4jw\"" Oct 30 02:13:52.266655 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.266622 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj"] Oct 30 02:13:52.287442 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.287402 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:52.355009 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.354975 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-istio-data\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.358279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.358239 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af31090b-f48b-4675-9add-665d0f5d42af-istio-podinfo\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.358492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.358289 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af31090b-f48b-4675-9add-665d0f5d42af-istiod-ca-cert\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.358492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.358386 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-workload-socket\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.358632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.358529 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-istio-envoy\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.358632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.358587 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87fr9\" (UniqueName: \"kubernetes.io/projected/af31090b-f48b-4675-9add-665d0f5d42af-kube-api-access-87fr9\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.358632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.358628 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af31090b-f48b-4675-9add-665d0f5d42af-istio-token\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.358799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.358666 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-workload-certs\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.358799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.358736 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-credential-socket\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.450160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.441545 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:52.464845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.460880 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-istio-data\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.464845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.460945 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af31090b-f48b-4675-9add-665d0f5d42af-istio-podinfo\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.464845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.460977 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af31090b-f48b-4675-9add-665d0f5d42af-istiod-ca-cert\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.464845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.461050 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-workload-socket\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.464845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.461253 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-istio-envoy\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.464845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.461311 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87fr9\" (UniqueName: \"kubernetes.io/projected/af31090b-f48b-4675-9add-665d0f5d42af-kube-api-access-87fr9\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.464845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.461351 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af31090b-f48b-4675-9add-665d0f5d42af-istio-token\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.464845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.461385 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-workload-certs\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.464845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.461443 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-credential-socket\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.464845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.461964 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-credential-socket\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.464845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.462231 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-istio-data\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.464845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.462904 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-workload-certs\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.468896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.468866 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af31090b-f48b-4675-9add-665d0f5d42af-istiod-ca-cert\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.469773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.469746 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-workload-socket\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.477642 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.477616 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af31090b-f48b-4675-9add-665d0f5d42af-istio-podinfo\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.482102 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.481446 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-istio-envoy\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.486791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.486761 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af31090b-f48b-4675-9add-665d0f5d42af-istio-token\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.488004 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.487950 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87fr9\" (UniqueName: \"kubernetes.io/projected/af31090b-f48b-4675-9add-665d0f5d42af-kube-api-access-87fr9\") pod \"e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.567948 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.567905 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:52.924311 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.924055 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj"] Oct 30 02:13:52.927466 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:13:52.927430 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf31090b_f48b_4675_9add_665d0f5d42af.slice/crio-26c5f0ee224c1d84f1043fd456925a40caf831d1394dd78a4eb726ec57e5a168 WatchSource:0}: Error finding container 26c5f0ee224c1d84f1043fd456925a40caf831d1394dd78a4eb726ec57e5a168: Status 404 returned error can't find the container with id 26c5f0ee224c1d84f1043fd456925a40caf831d1394dd78a4eb726ec57e5a168 Oct 30 02:13:52.930182 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.930147 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:13:52.930308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.930234 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:13:52.930308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.930276 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:13:52.998300 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:52.998263 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" event={"ID":"af31090b-f48b-4675-9add-665d0f5d42af","Type":"ContainerStarted","Data":"26c5f0ee224c1d84f1043fd456925a40caf831d1394dd78a4eb726ec57e5a168"} Oct 30 02:13:53.471002 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:53.470935 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:53.575698 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:53.575648 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:53.696720 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:53.696674 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:54.006548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.006509 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" event={"ID":"af31090b-f48b-4675-9add-665d0f5d42af","Type":"ContainerStarted","Data":"e9dc92290e7b950f887056916718f686a3923657c26628f386896b13c4723dec"} Oct 30 02:13:54.059775 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.059722 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" podStartSLOduration=2.059708552 podStartE2EDuration="2.059708552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:54.058295282 +0000 UTC m=+2663.637086388" watchObservedRunningTime="2026-10-30 02:13:54.059708552 +0000 UTC m=+2663.638499633" Oct 30 02:13:54.138832 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.138800 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:54.568234 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.568205 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:54.574280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.574254 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:54.612701 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.612672 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9"] Oct 30 02:13:54.617311 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.617286 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.632261 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.632228 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-a0684f-openshift-default-dockercfg-t857n\"" Oct 30 02:13:54.656058 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.656019 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9"] Oct 30 02:13:54.696275 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.696235 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k94w\" (UniqueName: \"kubernetes.io/projected/563a9420-524b-48cc-9e77-2b6001604841-kube-api-access-9k94w\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.696479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.696360 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/563a9420-524b-48cc-9e77-2b6001604841-istio-token\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.696479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.696400 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-workload-certs\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.696616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.696511 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-workload-socket\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.696616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.696544 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/563a9420-524b-48cc-9e77-2b6001604841-istiod-ca-cert\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.696616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.696569 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-istio-data\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.696616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.696599 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-credential-socket\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.696840 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.696682 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/563a9420-524b-48cc-9e77-2b6001604841-istio-podinfo\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.696840 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.696712 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-istio-envoy\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.723477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.723444 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:54.809127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.800949 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k94w\" (UniqueName: \"kubernetes.io/projected/563a9420-524b-48cc-9e77-2b6001604841-kube-api-access-9k94w\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.809127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.801039 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/563a9420-524b-48cc-9e77-2b6001604841-istio-token\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.809127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.801084 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-workload-certs\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.809127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.801222 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-workload-socket\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.809127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.801261 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/563a9420-524b-48cc-9e77-2b6001604841-istiod-ca-cert\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.809127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.801291 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-istio-data\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.809127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.801321 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-credential-socket\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.809127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.801409 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/563a9420-524b-48cc-9e77-2b6001604841-istio-podinfo\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.809127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.801439 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-istio-envoy\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.809127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.802209 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-workload-socket\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.809127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.802965 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-workload-certs\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.809127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.803237 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-istio-data\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.809127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.803850 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/563a9420-524b-48cc-9e77-2b6001604841-istiod-ca-cert\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.809127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.804102 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-credential-socket\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.826577 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.826503 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-istio-envoy\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.843189 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.843149 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/563a9420-524b-48cc-9e77-2b6001604841-istio-podinfo\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.853957 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.853922 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k94w\" (UniqueName: \"kubernetes.io/projected/563a9420-524b-48cc-9e77-2b6001604841-kube-api-access-9k94w\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.855651 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.855619 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:54.856720 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.856696 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/563a9420-524b-48cc-9e77-2b6001604841-istio-token\") pod \"e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:54.936112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:54.935702 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:55.016183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:55.016150 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:55.025152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:55.024736 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:13:55.255777 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:13:55.255731 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:13:55.289066 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:55.289034 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9"] Oct 30 02:13:55.290816 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:55.290791 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:55.293102 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:13:55.293059 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563a9420_524b_48cc_9e77_2b6001604841.slice/crio-6287334840b2171729c721739124398b2bc95568d888a440566cb0426724fd4f WatchSource:0}: Error finding container 6287334840b2171729c721739124398b2bc95568d888a440566cb0426724fd4f: Status 404 returned error can't find the container with id 6287334840b2171729c721739124398b2bc95568d888a440566cb0426724fd4f Oct 30 02:13:55.296290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:55.296251 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:13:55.296384 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:55.296363 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:13:55.296425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:55.296408 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:13:55.396988 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:55.396953 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:55.511106 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:55.510944 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:56.024848 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:56.024806 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" event={"ID":"563a9420-524b-48cc-9e77-2b6001604841","Type":"ContainerStarted","Data":"2356216a486173038d8e3dc5102885229785d20bddb29cf0ee8de60ee884bc81"} Oct 30 02:13:56.025388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:56.024873 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" event={"ID":"563a9420-524b-48cc-9e77-2b6001604841","Type":"ContainerStarted","Data":"6287334840b2171729c721739124398b2bc95568d888a440566cb0426724fd4f"} Oct 30 02:13:56.068155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:56.068104 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" podStartSLOduration=2.068076119 podStartE2EDuration="2.068076119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:56.067024777 +0000 UTC m=+2665.645815859" watchObservedRunningTime="2026-10-30 02:13:56.068076119 +0000 UTC m=+2665.646867201" Oct 30 02:13:56.469491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:56.469453 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:56.592826 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:56.592783 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:56.936578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:56.936548 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:56.944056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:56.944026 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:57.032120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:57.032057 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:57.032864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:57.032843 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:13:57.517713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:57.517673 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:58.540775 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:58.540732 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:58.658051 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:58.658014 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:58.771996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:58.771966 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:59.291151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.291104 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" podUID="0cff25a3-26e7-423f-a856-2568ebb922ca" containerName="main" containerID="cri-o://764670faa98c76e9528660a8bbd4b2d874b0804de3f07fd927adaab1f344af1b" gracePeriod=45 Oct 30 02:13:59.498798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.498685 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:13:59.580765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.580732 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:13:59.604923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.604895 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-dshm\") pod \"0cff25a3-26e7-423f-a856-2568ebb922ca\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " Oct 30 02:13:59.604923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.604928 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-home\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-home\") pod \"0cff25a3-26e7-423f-a856-2568ebb922ca\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " Oct 30 02:13:59.605198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.605030 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-tmp-dir\") pod \"0cff25a3-26e7-423f-a856-2568ebb922ca\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " Oct 30 02:13:59.605198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.605056 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cff25a3-26e7-423f-a856-2568ebb922ca-kube-api-access-vbsw8\" (UniqueName: \"kubernetes.io/projected/0cff25a3-26e7-423f-a856-2568ebb922ca-kube-api-access-vbsw8\") pod \"0cff25a3-26e7-423f-a856-2568ebb922ca\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " Oct 30 02:13:59.605198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.605079 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-model-cache\") pod \"0cff25a3-26e7-423f-a856-2568ebb922ca\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " Oct 30 02:13:59.605198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.605165 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cff25a3-26e7-423f-a856-2568ebb922ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/0cff25a3-26e7-423f-a856-2568ebb922ca-tls-certs\") pod \"0cff25a3-26e7-423f-a856-2568ebb922ca\" (UID: \"0cff25a3-26e7-423f-a856-2568ebb922ca\") " Oct 30 02:13:59.605356 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.605295 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-home" pod "0cff25a3-26e7-423f-a856-2568ebb922ca" (UID: "0cff25a3-26e7-423f-a856-2568ebb922ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:59.605708 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.605681 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-tmp-dir" pod "0cff25a3-26e7-423f-a856-2568ebb922ca" (UID: "0cff25a3-26e7-423f-a856-2568ebb922ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:59.605859 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.605827 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:59.605859 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.605843 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:59.606013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.605889 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-model-cache" pod "0cff25a3-26e7-423f-a856-2568ebb922ca" (UID: "0cff25a3-26e7-423f-a856-2568ebb922ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:59.608004 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.607975 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-dshm" pod "0cff25a3-26e7-423f-a856-2568ebb922ca" (UID: "0cff25a3-26e7-423f-a856-2568ebb922ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:59.608188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.608067 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cff25a3-26e7-423f-a856-2568ebb922ca-kube-api-access-vbsw8" pod "0cff25a3-26e7-423f-a856-2568ebb922ca" (UID: "0cff25a3-26e7-423f-a856-2568ebb922ca"). InnerVolumeSpecName "kube-api-access-vbsw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:59.608188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.608151 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cff25a3-26e7-423f-a856-2568ebb922ca-tls-certs" pod "0cff25a3-26e7-423f-a856-2568ebb922ca" (UID: "0cff25a3-26e7-423f-a856-2568ebb922ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:13:59.707342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.707307 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbsw8\" (UniqueName: \"kubernetes.io/projected/0cff25a3-26e7-423f-a856-2568ebb922ca-kube-api-access-vbsw8\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:59.707342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.707339 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:59.707342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.707348 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cff25a3-26e7-423f-a856-2568ebb922ca-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:13:59.707583 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:13:59.707356 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cff25a3-26e7-423f-a856-2568ebb922ca-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:14:00.055846 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.055809 5998 generic.go:356] "Generic (PLEG): container finished" podID="0cff25a3-26e7-423f-a856-2568ebb922ca" containerID="764670faa98c76e9528660a8bbd4b2d874b0804de3f07fd927adaab1f344af1b" exitCode=0 Oct 30 02:14:00.056036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.055852 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" event={"ID":"0cff25a3-26e7-423f-a856-2568ebb922ca","Type":"ContainerDied","Data":"764670faa98c76e9528660a8bbd4b2d874b0804de3f07fd927adaab1f344af1b"} Oct 30 02:14:00.056036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.055864 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" Oct 30 02:14:00.056036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.055884 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np" event={"ID":"0cff25a3-26e7-423f-a856-2568ebb922ca","Type":"ContainerDied","Data":"e5d1a8d47dcb88ce7593ec1682a8fd42fc5ae4502028d331f2a2303026273a41"} Oct 30 02:14:00.056036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.055901 5998 scope.go:122] "RemoveContainer" containerID="764670faa98c76e9528660a8bbd4b2d874b0804de3f07fd927adaab1f344af1b" Oct 30 02:14:00.067613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.067594 5998 scope.go:122] "RemoveContainer" containerID="764670faa98c76e9528660a8bbd4b2d874b0804de3f07fd927adaab1f344af1b" Oct 30 02:14:00.067890 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:14:00.067866 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764670faa98c76e9528660a8bbd4b2d874b0804de3f07fd927adaab1f344af1b\": container with ID starting with 764670faa98c76e9528660a8bbd4b2d874b0804de3f07fd927adaab1f344af1b not found: ID does not exist" containerID="764670faa98c76e9528660a8bbd4b2d874b0804de3f07fd927adaab1f344af1b" Oct 30 02:14:00.067977 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.067899 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764670faa98c76e9528660a8bbd4b2d874b0804de3f07fd927adaab1f344af1b"} err="failed to get container status \"764670faa98c76e9528660a8bbd4b2d874b0804de3f07fd927adaab1f344af1b\": rpc error: code = NotFound desc = could not find container \"764670faa98c76e9528660a8bbd4b2d874b0804de3f07fd927adaab1f344af1b\": container with ID starting with 764670faa98c76e9528660a8bbd4b2d874b0804de3f07fd927adaab1f344af1b not found: ID does not exist" Oct 30 02:14:00.104903 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.104866 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np"] Oct 30 02:14:00.114897 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.114864 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-24fb97/auth-test-model-24fb97-kserve-d69bffd77-468np"] Oct 30 02:14:00.349419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.349302 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:00.461126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.461074 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:00.469898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.469869 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b"] Oct 30 02:14:00.470422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.470407 5998 cpu_manager.go:414] "containerMap: removing container" podUID="0cff25a3-26e7-423f-a856-2568ebb922ca" containerName="main" Oct 30 02:14:00.470493 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.470423 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cff25a3-26e7-423f-a856-2568ebb922ca" containerName="main" Oct 30 02:14:00.470556 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.470547 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cff25a3-26e7-423f-a856-2568ebb922ca" containerName="main" Oct 30 02:14:00.475074 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.475051 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b" Oct 30 02:14:00.488035 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.488011 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-a0684f\"" Oct 30 02:14:00.488194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.488053 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-a0684f-dockercfg-znbbw\"" Oct 30 02:14:00.495843 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.495813 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b"] Oct 30 02:14:00.519526 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.519491 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dfbbf9-122b-49ed-abab-a9c107c457ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b\" (UID: \"85dfbbf9-122b-49ed-abab-a9c107c457ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b" Oct 30 02:14:00.519748 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.519725 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzfpr\" (UniqueName: \"kubernetes.io/projected/85dfbbf9-122b-49ed-abab-a9c107c457ab-kube-api-access-hzfpr\") pod \"payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b\" (UID: \"85dfbbf9-122b-49ed-abab-a9c107c457ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b" Oct 30 02:14:00.543789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.543746 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cff25a3-26e7-423f-a856-2568ebb922ca" path="/var/lib/kubelet/pods/0cff25a3-26e7-423f-a856-2568ebb922ca/volumes" Oct 30 02:14:00.623192 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.620988 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzfpr\" (UniqueName: \"kubernetes.io/projected/85dfbbf9-122b-49ed-abab-a9c107c457ab-kube-api-access-hzfpr\") pod \"payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b\" (UID: \"85dfbbf9-122b-49ed-abab-a9c107c457ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b" Oct 30 02:14:00.623192 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.621146 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dfbbf9-122b-49ed-abab-a9c107c457ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b\" (UID: \"85dfbbf9-122b-49ed-abab-a9c107c457ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b" Oct 30 02:14:00.623192 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.622370 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dfbbf9-122b-49ed-abab-a9c107c457ab-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b\" (UID: \"85dfbbf9-122b-49ed-abab-a9c107c457ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b" Oct 30 02:14:00.681114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.680893 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzfpr\" (UniqueName: \"kubernetes.io/projected/85dfbbf9-122b-49ed-abab-a9c107c457ab-kube-api-access-hzfpr\") pod \"payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b\" (UID: \"85dfbbf9-122b-49ed-abab-a9c107c457ab\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b" Oct 30 02:14:00.789428 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:00.789390 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b" Oct 30 02:14:01.099783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:01.099753 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b"] Oct 30 02:14:01.103773 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:14:01.103738 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85dfbbf9_122b_49ed_abab_a9c107c457ab.slice/crio-0d5b00583ed1f9d92aad1247a200faf2d0a948119a09cf6d1e0936278f4d7dc2 WatchSource:0}: Error finding container 0d5b00583ed1f9d92aad1247a200faf2d0a948119a09cf6d1e0936278f4d7dc2: Status 404 returned error can't find the container with id 0d5b00583ed1f9d92aad1247a200faf2d0a948119a09cf6d1e0936278f4d7dc2 Oct 30 02:14:01.106269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:01.106246 5998 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:14:01.266285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:01.266250 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:02.072871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:02.072826 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b" event={"ID":"85dfbbf9-122b-49ed-abab-a9c107c457ab","Type":"ContainerStarted","Data":"0d5b00583ed1f9d92aad1247a200faf2d0a948119a09cf6d1e0936278f4d7dc2"} Oct 30 02:14:02.760015 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:02.759972 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:04.091911 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:04.091870 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b" event={"ID":"85dfbbf9-122b-49ed-abab-a9c107c457ab","Type":"ContainerStarted","Data":"3c9155f55f4e3282f259398678f1f6cc1526844abafd39914f964efb6a0998a4"} Oct 30 02:14:04.092425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:04.091954 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b" Oct 30 02:14:04.130533 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:04.130470 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b" podStartSLOduration=1.8695281179999998 podStartE2EDuration="4.130445889s" totalImagesPullingTime="2.260917771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:14:04.128960702 +0000 UTC m=+2673.707751784" watchObservedRunningTime="2026-10-30 02:14:04.130445889 +0000 UTC m=+2673.709236970" Oct 30 02:14:04.258868 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:04.258833 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:05.418495 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.418412 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" podUID="696d7d69-7253-4856-b4bd-f0f650c0f2ce" containerName="main" containerID="cri-o://e7d6d1a73d0835013ef3b126d50516286825b69eea221330d8506e314e008950" gracePeriod=45 Oct 30 02:14:05.435424 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.435383 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" podUID="696d7d69-7253-4856-b4bd-f0f650c0f2ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.163:8000/ready\": dial tcp 10.133.0.163:8000: connect: connection refused" Oct 30 02:14:05.827222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.827188 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:14:05.883667 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.883634 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-home\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-home\") pod \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " Oct 30 02:14:05.883877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.883681 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/696d7d69-7253-4856-b4bd-f0f650c0f2ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/696d7d69-7253-4856-b4bd-f0f650c0f2ce-tls-certs\") pod \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " Oct 30 02:14:05.883877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.883725 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-dshm\") pod \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " Oct 30 02:14:05.883877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.883791 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/696d7d69-7253-4856-b4bd-f0f650c0f2ce-kube-api-access-kxwvq\" (UniqueName: \"kubernetes.io/projected/696d7d69-7253-4856-b4bd-f0f650c0f2ce-kube-api-access-kxwvq\") pod \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " Oct 30 02:14:05.883877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.883843 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-model-cache\") pod \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " Oct 30 02:14:05.884054 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.883900 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-tmp-dir\") pod \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\" (UID: \"696d7d69-7253-4856-b4bd-f0f650c0f2ce\") " Oct 30 02:14:05.884054 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.883953 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-home" pod "696d7d69-7253-4856-b4bd-f0f650c0f2ce" (UID: "696d7d69-7253-4856-b4bd-f0f650c0f2ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:05.884471 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.884435 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-model-cache" pod "696d7d69-7253-4856-b4bd-f0f650c0f2ce" (UID: "696d7d69-7253-4856-b4bd-f0f650c0f2ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:05.884598 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.884567 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-tmp-dir" pod "696d7d69-7253-4856-b4bd-f0f650c0f2ce" (UID: "696d7d69-7253-4856-b4bd-f0f650c0f2ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:05.884731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.884659 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:14:05.884731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.884713 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:14:05.886602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.886581 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-dshm" pod "696d7d69-7253-4856-b4bd-f0f650c0f2ce" (UID: "696d7d69-7253-4856-b4bd-f0f650c0f2ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:05.886733 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.886627 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/696d7d69-7253-4856-b4bd-f0f650c0f2ce-kube-api-access-kxwvq" pod "696d7d69-7253-4856-b4bd-f0f650c0f2ce" (UID: "696d7d69-7253-4856-b4bd-f0f650c0f2ce"). InnerVolumeSpecName "kube-api-access-kxwvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:05.886733 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.886695 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696d7d69-7253-4856-b4bd-f0f650c0f2ce-tls-certs" pod "696d7d69-7253-4856-b4bd-f0f650c0f2ce" (UID: "696d7d69-7253-4856-b4bd-f0f650c0f2ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:14:05.904175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.904144 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:05.985551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.985514 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/696d7d69-7253-4856-b4bd-f0f650c0f2ce-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:14:05.985551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.985548 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:14:05.985817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.985563 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxwvq\" (UniqueName: \"kubernetes.io/projected/696d7d69-7253-4856-b4bd-f0f650c0f2ce-kube-api-access-kxwvq\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:14:05.985817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:05.985577 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/696d7d69-7253-4856-b4bd-f0f650c0f2ce-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:14:06.118200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:06.111924 5998 generic.go:356] "Generic (PLEG): container finished" podID="696d7d69-7253-4856-b4bd-f0f650c0f2ce" containerID="e7d6d1a73d0835013ef3b126d50516286825b69eea221330d8506e314e008950" exitCode=0 Oct 30 02:14:06.118200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:06.111984 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" Oct 30 02:14:06.118200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:06.111984 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" event={"ID":"696d7d69-7253-4856-b4bd-f0f650c0f2ce","Type":"ContainerDied","Data":"e7d6d1a73d0835013ef3b126d50516286825b69eea221330d8506e314e008950"} Oct 30 02:14:06.118200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:06.112133 5998 scope.go:122] "RemoveContainer" containerID="e7d6d1a73d0835013ef3b126d50516286825b69eea221330d8506e314e008950" Oct 30 02:14:06.118200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:06.112302 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl" event={"ID":"696d7d69-7253-4856-b4bd-f0f650c0f2ce","Type":"ContainerDied","Data":"9bfea2120d8b9b7437b3bb2c122b85f775355359eeb70de8393c0abf89699a90"} Oct 30 02:14:06.131202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:06.131184 5998 scope.go:122] "RemoveContainer" containerID="e7d6d1a73d0835013ef3b126d50516286825b69eea221330d8506e314e008950" Oct 30 02:14:06.131521 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:14:06.131502 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d6d1a73d0835013ef3b126d50516286825b69eea221330d8506e314e008950\": container with ID starting with e7d6d1a73d0835013ef3b126d50516286825b69eea221330d8506e314e008950 not found: ID does not exist" containerID="e7d6d1a73d0835013ef3b126d50516286825b69eea221330d8506e314e008950" Oct 30 02:14:06.131613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:06.131529 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d6d1a73d0835013ef3b126d50516286825b69eea221330d8506e314e008950"} err="failed to get container status \"e7d6d1a73d0835013ef3b126d50516286825b69eea221330d8506e314e008950\": rpc error: code = NotFound desc = could not find container \"e7d6d1a73d0835013ef3b126d50516286825b69eea221330d8506e314e008950\": container with ID starting with e7d6d1a73d0835013ef3b126d50516286825b69eea221330d8506e314e008950 not found: ID does not exist" Oct 30 02:14:06.158683 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:06.158648 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl"] Oct 30 02:14:06.168972 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:06.168938 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-594253/auth-test-model-594253-kserve-5c5cc6d8fc-bfjgl"] Oct 30 02:14:06.540737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:06.540691 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="696d7d69-7253-4856-b4bd-f0f650c0f2ce" path="/var/lib/kubelet/pods/696d7d69-7253-4856-b4bd-f0f650c0f2ce/volumes" Oct 30 02:14:07.551898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:07.551856 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:09.129135 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:09.129084 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:10.602609 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:10.602573 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:10.866176 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:10.861070 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:14:11.060148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:11.060113 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:12.373907 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:12.373824 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:12.529116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:12.524023 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:12.880481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:12.880440 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:12.899217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:12.899175 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h"] Oct 30 02:14:12.915116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:12.900341 5998 cpu_manager.go:414] "containerMap: removing container" podUID="696d7d69-7253-4856-b4bd-f0f650c0f2ce" containerName="main" Oct 30 02:14:12.915396 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:12.915368 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="696d7d69-7253-4856-b4bd-f0f650c0f2ce" containerName="main" Oct 30 02:14:12.916076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:12.916044 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="696d7d69-7253-4856-b4bd-f0f650c0f2ce" containerName="main" Oct 30 02:14:12.991987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:12.991942 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h"] Oct 30 02:14:12.997348 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:12.997246 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.040164 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.038369 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-93ec46-openshift-default-dockercfg-t7gzq\"" Oct 30 02:14:13.103114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.096501 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-token\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.103114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.096574 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8532131f-abe1-426d-9a42-36ad42fb7cdb-istiod-ca-cert\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.103114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.096639 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-podinfo\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.103114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.096730 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-credential-socket\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.103114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.096781 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-envoy\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.103114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.096849 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-workload-certs\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.103114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.096907 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-workload-socket\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.103114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.096983 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-data\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.103114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.097042 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbpbz\" (UniqueName: \"kubernetes.io/projected/8532131f-abe1-426d-9a42-36ad42fb7cdb-kube-api-access-sbpbz\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.254855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.254814 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbpbz\" (UniqueName: \"kubernetes.io/projected/8532131f-abe1-426d-9a42-36ad42fb7cdb-kube-api-access-sbpbz\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.281605 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.271500 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:13.287539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.287496 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-token\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.298066 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.298029 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8532131f-abe1-426d-9a42-36ad42fb7cdb-istiod-ca-cert\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.299849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.299762 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-podinfo\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.302363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.301654 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-credential-socket\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.311154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.300323 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8532131f-abe1-426d-9a42-36ad42fb7cdb-istiod-ca-cert\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.311154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.310764 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-credential-socket\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.312359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.312328 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-envoy\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.314317 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.314282 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-workload-certs\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.316300 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.316275 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-workload-socket\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.318399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.318375 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-data\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.335634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.317227 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-workload-certs\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.336381 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.318879 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-data\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.337150 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.316952 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-workload-socket\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.359269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.359235 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbpbz\" (UniqueName: \"kubernetes.io/projected/8532131f-abe1-426d-9a42-36ad42fb7cdb-kube-api-access-sbpbz\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.376152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.373921 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-podinfo\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.376641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.376620 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-token\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.392128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.391348 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-envoy\") pod \"e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:13.534710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:13.533369 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:14.147115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:14.143774 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:14.167114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:14.165277 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h"] Oct 30 02:14:14.179914 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:14:14.179882 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8532131f_abe1_426d_9a42_36ad42fb7cdb.slice/crio-82a991226617a7db43864bb8680e640f877b7b14311a32ce313518ecc6cec3db WatchSource:0}: Error finding container 82a991226617a7db43864bb8680e640f877b7b14311a32ce313518ecc6cec3db: Status 404 returned error can't find the container with id 82a991226617a7db43864bb8680e640f877b7b14311a32ce313518ecc6cec3db Oct 30 02:14:14.183723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:14.183683 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:14:14.184303 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:14.184277 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:14:14.184477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:14.184458 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:14:14.459647 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:14.459608 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" event={"ID":"8532131f-abe1-426d-9a42-36ad42fb7cdb","Type":"ContainerStarted","Data":"82a991226617a7db43864bb8680e640f877b7b14311a32ce313518ecc6cec3db"} Oct 30 02:14:15.117684 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:15.117641 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b" Oct 30 02:14:15.473608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:15.473572 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" event={"ID":"8532131f-abe1-426d-9a42-36ad42fb7cdb","Type":"ContainerStarted","Data":"3cd7d398c5dc61ee87456fa1d7adbb155d01ac3f9c9d421d2f637ac3c163a3c5"} Oct 30 02:14:15.540763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:15.540728 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:15.571733 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:15.571428 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:15.618113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:15.616886 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" podStartSLOduration=3.616866748 podStartE2EDuration="3.616866748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:14:15.531366911 +0000 UTC m=+2685.110157993" watchObservedRunningTime="2026-10-30 02:14:15.616866748 +0000 UTC m=+2685.195657830" Oct 30 02:14:15.797841 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:15.797760 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:16.480353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:16.480328 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:16.481221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:16.481199 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:14:17.365750 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:17.365715 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:17.882144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:17.879817 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:17.991817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:17.991782 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:19.275717 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:19.275682 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:19.670237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:19.670200 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:19.778330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:19.778295 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:20.982988 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:20.982954 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:22.551428 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:22.551394 5998 generic.go:356] "Generic (PLEG): container finished" podID="1411791d-2b8f-46e0-8b26-8799563d8c4b" containerID="c0b725d9ba4c863b51985264780bce7ae018b22fcf1c74e2af9608c380d8d1da" exitCode=0 Oct 30 02:14:22.553387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:22.553356 5998 generic.go:356] "Generic (PLEG): container finished" podID="57be1a46-2149-46f4-a048-b099d67d3a6b" containerID="122a350bf9d6bd79a9e50c83b560c5ddf2991e2b97e66d4f77477449c28424b6" exitCode=0 Oct 30 02:14:22.554013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:22.553986 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" event={"ID":"1411791d-2b8f-46e0-8b26-8799563d8c4b","Type":"ContainerDied","Data":"c0b725d9ba4c863b51985264780bce7ae018b22fcf1c74e2af9608c380d8d1da"} Oct 30 02:14:22.554065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:22.554054 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" event={"ID":"57be1a46-2149-46f4-a048-b099d67d3a6b","Type":"ContainerDied","Data":"122a350bf9d6bd79a9e50c83b560c5ddf2991e2b97e66d4f77477449c28424b6"} Oct 30 02:14:22.554404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:22.554389 5998 scope.go:122] "RemoveContainer" containerID="122a350bf9d6bd79a9e50c83b560c5ddf2991e2b97e66d4f77477449c28424b6" Oct 30 02:14:22.554534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:22.554518 5998 scope.go:122] "RemoveContainer" containerID="c0b725d9ba4c863b51985264780bce7ae018b22fcf1c74e2af9608c380d8d1da" Oct 30 02:14:22.556152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:22.556120 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:14:22.556241 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:22.556199 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:14:22.556241 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:22.556080 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:14:22.556315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:22.556242 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:14:22.556315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:22.556273 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:14:22.556383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:22.556316 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:14:22.670613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:22.670517 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:23.570984 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:23.570945 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" event={"ID":"57be1a46-2149-46f4-a048-b099d67d3a6b","Type":"ContainerStarted","Data":"32a424da9f0dbaaeef09ef6cd48a253ec4bc55e899d8fdebfeccac9e52eba361"} Oct 30 02:14:23.573361 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:23.573335 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" event={"ID":"1411791d-2b8f-46e0-8b26-8799563d8c4b","Type":"ContainerStarted","Data":"636c8cafb70c3d354c5985475af955149a1902504a3990d4722f7e3115f75136"} Oct 30 02:14:23.625116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:23.624228 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:14:23.642115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:23.626152 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 02:14:23.642115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:23.626209 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" podUID="57be1a46-2149-46f4-a048-b099d67d3a6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 02:14:23.680411 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:23.680375 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:14:23.681435 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:23.681405 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 02:14:23.681581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:23.681453 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" podUID="1411791d-2b8f-46e0-8b26-8799563d8c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 02:14:24.271123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:24.270388 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:24.399950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:24.399842 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:24.523171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:24.522779 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:24.620116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:24.615282 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:14:24.620116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:24.615706 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 02:14:24.620116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:24.615780 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" podUID="57be1a46-2149-46f4-a048-b099d67d3a6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 02:14:24.680346 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:24.680309 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 02:14:24.680532 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:24.680355 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" podUID="1411791d-2b8f-46e0-8b26-8799563d8c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 02:14:25.345626 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:14:25.345418 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:14:25.621179 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:25.613789 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 02:14:25.621179 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:25.613848 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" podUID="57be1a46-2149-46f4-a048-b099d67d3a6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 02:14:25.680344 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:25.680298 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 02:14:25.680535 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:25.680357 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" podUID="1411791d-2b8f-46e0-8b26-8799563d8c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 02:14:25.930684 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:25.930525 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:26.643114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:26.628191 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 02:14:26.643114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:26.628260 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" podUID="57be1a46-2149-46f4-a048-b099d67d3a6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 02:14:26.680785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:26.680750 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 02:14:26.681031 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:26.681004 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" podUID="1411791d-2b8f-46e0-8b26-8799563d8c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 02:14:26.780368 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:26.780337 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:14:27.490707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:27.490671 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:27.599325 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:27.599284 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:27.648249 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:27.646680 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 02:14:27.648249 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:27.646744 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" podUID="57be1a46-2149-46f4-a048-b099d67d3a6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 02:14:27.682649 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:27.682617 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 02:14:27.682982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:27.682934 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" podUID="1411791d-2b8f-46e0-8b26-8799563d8c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 02:14:28.615680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:28.615625 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 02:14:28.616028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:28.615977 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" podUID="57be1a46-2149-46f4-a048-b099d67d3a6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 02:14:28.680638 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:28.680597 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 02:14:28.681288 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:28.681232 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" podUID="1411791d-2b8f-46e0-8b26-8799563d8c4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 02:14:29.472525 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:29.472452 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:29.579111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:29.579070 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:29.685896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:29.685864 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:14:29.687178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:29.687153 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:14:29.728309 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:29.728188 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:14:29.729983 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:29.729959 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:14:30.778604 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:30.778563 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:31.075439 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.075316 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:31.175505 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.175455 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh"] Oct 30 02:14:31.180717 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.180691 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.223113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.206328 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0fdc4e-openshift-default-dockercfg-j4dkk\"" Oct 30 02:14:31.243069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.243034 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh"] Oct 30 02:14:31.243328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.243024 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:31.308060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.308021 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfnm7\" (UniqueName: \"kubernetes.io/projected/eef42836-3659-4fdc-b2ff-b8003e3e47a3-kube-api-access-bfnm7\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.332961 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.332861 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-credential-socket\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.333328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.333302 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istiod-ca-cert\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.333527 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.333510 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-podinfo\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.338794 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.338762 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-workload-certs\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.339069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.339051 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-workload-socket\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.339277 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.339259 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-envoy\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.339485 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.339469 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-token\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.342858 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.342832 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-data\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.371312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.371282 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:31.485491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.485440 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-workload-certs\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.489135 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.489103 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-workload-socket\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.491468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.491441 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-envoy\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.492068 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.492047 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-token\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.492669 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.492646 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-data\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.493585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.493566 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfnm7\" (UniqueName: \"kubernetes.io/projected/eef42836-3659-4fdc-b2ff-b8003e3e47a3-kube-api-access-bfnm7\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.493728 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.493711 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-credential-socket\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.494814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.494795 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istiod-ca-cert\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.497900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.490047 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-workload-socket\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.497900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.494403 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-data\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.497900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.494672 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-credential-socket\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.497900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.486231 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-workload-certs\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.497900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.495826 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istiod-ca-cert\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.498991 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.498963 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-podinfo\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.559597 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.554207 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfnm7\" (UniqueName: \"kubernetes.io/projected/eef42836-3659-4fdc-b2ff-b8003e3e47a3-kube-api-access-bfnm7\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.562767 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.562733 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-envoy\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.563728 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.563691 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-podinfo\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.564768 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.564717 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-token\") pod \"e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:31.838384 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:31.838327 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:32.177990 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:32.177958 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh"] Oct 30 02:14:32.179465 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:14:32.179438 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef42836_3659_4fdc_b2ff_b8003e3e47a3.slice/crio-c48cc7c3f1f631e2b2137c51a3100345b7dc37aa2dc71dfa5700ffac22d0b804 WatchSource:0}: Error finding container c48cc7c3f1f631e2b2137c51a3100345b7dc37aa2dc71dfa5700ffac22d0b804: Status 404 returned error can't find the container with id c48cc7c3f1f631e2b2137c51a3100345b7dc37aa2dc71dfa5700ffac22d0b804 Oct 30 02:14:32.183391 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:32.183353 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:14:32.183478 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:32.183443 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:14:32.183528 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:32.183486 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:14:32.664786 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:32.664738 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:32.737004 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:32.736968 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" event={"ID":"eef42836-3659-4fdc-b2ff-b8003e3e47a3","Type":"ContainerStarted","Data":"36eb90a21df0e911eec233e8ffa2718e87689656261903f0b350f12f26b420b4"} Oct 30 02:14:32.737239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:32.737037 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" event={"ID":"eef42836-3659-4fdc-b2ff-b8003e3e47a3","Type":"ContainerStarted","Data":"c48cc7c3f1f631e2b2137c51a3100345b7dc37aa2dc71dfa5700ffac22d0b804"} Oct 30 02:14:32.776044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:32.776008 5998 scope.go:122] "RemoveContainer" containerID="78db3a829cb93f277f8aa2f69dbe4096c421154b7ba9e412f5e23adb1b4eebd8" Oct 30 02:14:32.802957 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:32.802909 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" podStartSLOduration=1.802894345 podStartE2EDuration="1.802894345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:14:32.789209371 +0000 UTC m=+2702.368000453" watchObservedRunningTime="2026-10-30 02:14:32.802894345 +0000 UTC m=+2702.381685430" Oct 30 02:14:32.839458 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:32.839429 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:33.845409 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:33.845375 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:34.291707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:34.291667 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:34.750994 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:34.750959 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:34.752065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:34.752045 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:14:35.151865 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:35.151784 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:35.263687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:35.263643 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:36.076813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:36.076778 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:36.862416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:36.862376 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:36.974178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:36.974117 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:37.448844 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:37.448801 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:37.752679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:37.752578 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:39.381068 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:39.381028 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:40.965804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:40.965765 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:42.456818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:42.456738 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:43.959836 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:43.959797 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:45.587802 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:45.587766 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:47.053320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:47.053279 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:14:47.263446 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:47.263412 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:49.075778 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:49.075632 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:49.189157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:49.189071 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:49.295477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:49.294960 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:50.819650 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:50.819612 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:51.676772 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:51.676733 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11190-2f177e"] Oct 30 02:14:51.682244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:51.682221 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11190-2f177e" Oct 30 02:14:51.704335 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:51.704296 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11190-2f177e"] Oct 30 02:14:51.716115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:51.714306 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2sddb\"" Oct 30 02:14:51.767296 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:51.763345 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtz6w\" (UniqueName: \"kubernetes.io/projected/9bf4ba08-f29b-4778-a3ea-b2a0cea035ac-kube-api-access-rtz6w\") pod \"test-curl-11190-2f177e\" (UID: \"9bf4ba08-f29b-4778-a3ea-b2a0cea035ac\") " pod="opendatahub/test-curl-11190-2f177e" Oct 30 02:14:51.868265 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:51.868221 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtz6w\" (UniqueName: \"kubernetes.io/projected/9bf4ba08-f29b-4778-a3ea-b2a0cea035ac-kube-api-access-rtz6w\") pod \"test-curl-11190-2f177e\" (UID: \"9bf4ba08-f29b-4778-a3ea-b2a0cea035ac\") " pod="opendatahub/test-curl-11190-2f177e" Oct 30 02:14:51.890853 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:51.890820 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtz6w\" (UniqueName: \"kubernetes.io/projected/9bf4ba08-f29b-4778-a3ea-b2a0cea035ac-kube-api-access-rtz6w\") pod \"test-curl-11190-2f177e\" (UID: \"9bf4ba08-f29b-4778-a3ea-b2a0cea035ac\") " pod="opendatahub/test-curl-11190-2f177e" Oct 30 02:14:51.995744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:51.995654 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11190-2f177e" Oct 30 02:14:52.467751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:52.467686 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11190-2f177e"] Oct 30 02:14:52.480672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:52.480647 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:52.980814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:52.980771 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11190-2f177e" event={"ID":"9bf4ba08-f29b-4778-a3ea-b2a0cea035ac","Type":"ContainerStarted","Data":"feb5a0e933e48e722fafce9aed20615a6bb5d6931622874800bfc2bc43d2a273"} Oct 30 02:14:52.980814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:52.980816 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11190-2f177e" event={"ID":"9bf4ba08-f29b-4778-a3ea-b2a0cea035ac","Type":"ContainerStarted","Data":"241dfac76fd6d92cda59151f5b0cf707447ec9c5ecc192b255411e0f53a4bf0a"} Oct 30 02:14:53.957115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:53.954889 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11190-2f177e" podStartSLOduration=2.954867083 podStartE2EDuration="2.954867083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:14:53.029284539 +0000 UTC m=+2722.608075624" watchObservedRunningTime="2026-10-30 02:14:53.954867083 +0000 UTC m=+2723.533658164" Oct 30 02:14:53.963973 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:53.962307 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11190-2f177e"] Oct 30 02:14:53.968110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:53.968060 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11190-2f177e"] Oct 30 02:14:54.046114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.035685 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:54.179419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.179382 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4"] Oct 30 02:14:54.180224 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.180204 5998 cpu_manager.go:414] "containerMap: removing container" podUID="9bf4ba08-f29b-4778-a3ea-b2a0cea035ac" containerName="test-curl-11190-2f177e" Oct 30 02:14:54.180341 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.180226 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf4ba08-f29b-4778-a3ea-b2a0cea035ac" containerName="test-curl-11190-2f177e" Oct 30 02:14:54.180403 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.180394 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf4ba08-f29b-4778-a3ea-b2a0cea035ac" containerName="test-curl-11190-2f177e" Oct 30 02:14:54.184577 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.184554 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4" Oct 30 02:14:54.197657 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.197503 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:14:54.197657 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.197568 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:14:54.197657 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.197637 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:14:54.212164 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.200353 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4"] Oct 30 02:14:54.237115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.232701 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:14:54.289136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.288730 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcwv2\" (UniqueName: \"kubernetes.io/projected/fc72bfa4-a197-49da-803b-1336391e40f8-kube-api-access-mcwv2\") pod \"maas-api-revoke-sub-4877029e-j8vk4\" (UID: \"fc72bfa4-a197-49da-803b-1336391e40f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4" Oct 30 02:14:54.289136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.288856 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc72bfa4-a197-49da-803b-1336391e40f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4877029e-j8vk4\" (UID: \"fc72bfa4-a197-49da-803b-1336391e40f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4" Oct 30 02:14:54.289136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.288939 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc72bfa4-a197-49da-803b-1336391e40f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4877029e-j8vk4\" (UID: \"fc72bfa4-a197-49da-803b-1336391e40f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4" Oct 30 02:14:54.389765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.389728 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcwv2\" (UniqueName: \"kubernetes.io/projected/fc72bfa4-a197-49da-803b-1336391e40f8-kube-api-access-mcwv2\") pod \"maas-api-revoke-sub-4877029e-j8vk4\" (UID: \"fc72bfa4-a197-49da-803b-1336391e40f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4" Oct 30 02:14:54.389958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.389808 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc72bfa4-a197-49da-803b-1336391e40f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4877029e-j8vk4\" (UID: \"fc72bfa4-a197-49da-803b-1336391e40f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4" Oct 30 02:14:54.389958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.389876 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc72bfa4-a197-49da-803b-1336391e40f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4877029e-j8vk4\" (UID: \"fc72bfa4-a197-49da-803b-1336391e40f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4" Oct 30 02:14:54.390960 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.390940 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc72bfa4-a197-49da-803b-1336391e40f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4877029e-j8vk4\" (UID: \"fc72bfa4-a197-49da-803b-1336391e40f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4" Oct 30 02:14:54.392417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.392396 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc72bfa4-a197-49da-803b-1336391e40f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4877029e-j8vk4\" (UID: \"fc72bfa4-a197-49da-803b-1336391e40f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4" Oct 30 02:14:54.448922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.448836 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcwv2\" (UniqueName: \"kubernetes.io/projected/fc72bfa4-a197-49da-803b-1336391e40f8-kube-api-access-mcwv2\") pod \"maas-api-revoke-sub-4877029e-j8vk4\" (UID: \"fc72bfa4-a197-49da-803b-1336391e40f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4" Oct 30 02:14:54.499182 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.499099 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4" Oct 30 02:14:54.852056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:54.852010 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4"] Oct 30 02:14:54.855604 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:14:54.855555 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc72bfa4_a197_49da_803b_1336391e40f8.slice/crio-6fba2da36a5cfc90677634472ec986b30ba109c8c4d7668e95bc75d33b8abb3a WatchSource:0}: Error finding container 6fba2da36a5cfc90677634472ec986b30ba109c8c4d7668e95bc75d33b8abb3a: Status 404 returned error can't find the container with id 6fba2da36a5cfc90677634472ec986b30ba109c8c4d7668e95bc75d33b8abb3a Oct 30 02:14:55.001344 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:55.001301 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11190-2f177e" podUID="9bf4ba08-f29b-4778-a3ea-b2a0cea035ac" containerName="test-curl-11190-2f177e" containerID="cri-o://feb5a0e933e48e722fafce9aed20615a6bb5d6931622874800bfc2bc43d2a273" gracePeriod=2 Oct 30 02:14:55.001525 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:55.001441 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4" event={"ID":"fc72bfa4-a197-49da-803b-1336391e40f8","Type":"ContainerStarted","Data":"6fba2da36a5cfc90677634472ec986b30ba109c8c4d7668e95bc75d33b8abb3a"} Oct 30 02:14:55.230281 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:14:55.230241 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:14:55.574567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:55.574471 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:56.018110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:56.015212 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4" event={"ID":"fc72bfa4-a197-49da-803b-1336391e40f8","Type":"ContainerStarted","Data":"58e57e08c9d5dad7471ef7b0b26fe455e80e6e6a5286dc260d2ef3620a50d8e9"} Oct 30 02:14:56.058148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:56.056981 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4" podStartSLOduration=2.056961286 podStartE2EDuration="2.056961286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:14:56.054376761 +0000 UTC m=+2725.633167843" watchObservedRunningTime="2026-10-30 02:14:56.056961286 +0000 UTC m=+2725.635752368" Oct 30 02:14:56.058370 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:56.058269 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:14:56.058435 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:56.058373 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" podUID="af31090b-f48b-4675-9add-665d0f5d42af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:14:57.027672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:57.027632 5998 generic.go:356] "Generic (PLEG): container finished" podID="fc72bfa4-a197-49da-803b-1336391e40f8" containerID="58e57e08c9d5dad7471ef7b0b26fe455e80e6e6a5286dc260d2ef3620a50d8e9" exitCode=0 Oct 30 02:14:57.028070 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:57.027711 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4" event={"ID":"fc72bfa4-a197-49da-803b-1336391e40f8","Type":"ContainerDied","Data":"58e57e08c9d5dad7471ef7b0b26fe455e80e6e6a5286dc260d2ef3620a50d8e9"} Oct 30 02:14:57.198028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:57.197990 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:57.276477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:57.269710 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11190-2f177e" Oct 30 02:14:57.358069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:57.358035 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf4ba08-f29b-4778-a3ea-b2a0cea035ac-kube-api-access-rtz6w\" (UniqueName: \"kubernetes.io/projected/9bf4ba08-f29b-4778-a3ea-b2a0cea035ac-kube-api-access-rtz6w\") pod \"9bf4ba08-f29b-4778-a3ea-b2a0cea035ac\" (UID: \"9bf4ba08-f29b-4778-a3ea-b2a0cea035ac\") " Oct 30 02:14:57.360237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:57.360206 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf4ba08-f29b-4778-a3ea-b2a0cea035ac-kube-api-access-rtz6w" pod "9bf4ba08-f29b-4778-a3ea-b2a0cea035ac" (UID: "9bf4ba08-f29b-4778-a3ea-b2a0cea035ac"). InnerVolumeSpecName "kube-api-access-rtz6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:57.460759 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:57.460720 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtz6w\" (UniqueName: \"kubernetes.io/projected/9bf4ba08-f29b-4778-a3ea-b2a0cea035ac-kube-api-access-rtz6w\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.041935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.041891 5998 generic.go:356] "Generic (PLEG): container finished" podID="9bf4ba08-f29b-4778-a3ea-b2a0cea035ac" containerID="feb5a0e933e48e722fafce9aed20615a6bb5d6931622874800bfc2bc43d2a273" exitCode=137 Oct 30 02:14:58.042463 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.042225 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11190-2f177e" Oct 30 02:14:58.045214 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.045165 5998 scope.go:122] "RemoveContainer" containerID="feb5a0e933e48e722fafce9aed20615a6bb5d6931622874800bfc2bc43d2a273" Oct 30 02:14:58.058900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.058882 5998 scope.go:122] "RemoveContainer" containerID="feb5a0e933e48e722fafce9aed20615a6bb5d6931622874800bfc2bc43d2a273" Oct 30 02:14:58.059295 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:14:58.059273 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb5a0e933e48e722fafce9aed20615a6bb5d6931622874800bfc2bc43d2a273\": container with ID starting with feb5a0e933e48e722fafce9aed20615a6bb5d6931622874800bfc2bc43d2a273 not found: ID does not exist" containerID="feb5a0e933e48e722fafce9aed20615a6bb5d6931622874800bfc2bc43d2a273" Oct 30 02:14:58.059383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.059313 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb5a0e933e48e722fafce9aed20615a6bb5d6931622874800bfc2bc43d2a273"} err="failed to get container status \"feb5a0e933e48e722fafce9aed20615a6bb5d6931622874800bfc2bc43d2a273\": rpc error: code = NotFound desc = could not find container \"feb5a0e933e48e722fafce9aed20615a6bb5d6931622874800bfc2bc43d2a273\": container with ID starting with feb5a0e933e48e722fafce9aed20615a6bb5d6931622874800bfc2bc43d2a273 not found: ID does not exist" Oct 30 02:14:58.066017 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:14:58.065977 5998 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11190-2f177e\" is forbidden: User \"system:node:ip-10-0-130-0.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-0.ec2.internal' and this object" podUID="9bf4ba08-f29b-4778-a3ea-b2a0cea035ac" pod="opendatahub/test-curl-11190-2f177e" Oct 30 02:14:58.073248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.073218 5998 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:14:58.073397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.073264 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" podUID="563a9420-524b-48cc-9e77-2b6001604841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:14:58.238574 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.238541 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4" Oct 30 02:14:58.250470 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:14:58.250423 5998 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11190-2f177e\" is forbidden: User \"system:node:ip-10-0-130-0.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-0.ec2.internal' and this object" podUID="9bf4ba08-f29b-4778-a3ea-b2a0cea035ac" pod="opendatahub/test-curl-11190-2f177e" Oct 30 02:14:58.371458 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.371423 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc72bfa4-a197-49da-803b-1336391e40f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc72bfa4-a197-49da-803b-1336391e40f8-maas-api-openshift-ca\") pod \"fc72bfa4-a197-49da-803b-1336391e40f8\" (UID: \"fc72bfa4-a197-49da-803b-1336391e40f8\") " Oct 30 02:14:58.371663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.371644 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc72bfa4-a197-49da-803b-1336391e40f8-kube-api-access-mcwv2\" (UniqueName: \"kubernetes.io/projected/fc72bfa4-a197-49da-803b-1336391e40f8-kube-api-access-mcwv2\") pod \"fc72bfa4-a197-49da-803b-1336391e40f8\" (UID: \"fc72bfa4-a197-49da-803b-1336391e40f8\") " Oct 30 02:14:58.371778 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.371759 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc72bfa4-a197-49da-803b-1336391e40f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc72bfa4-a197-49da-803b-1336391e40f8-maas-api-xks-ca\") pod \"fc72bfa4-a197-49da-803b-1336391e40f8\" (UID: \"fc72bfa4-a197-49da-803b-1336391e40f8\") " Oct 30 02:14:58.371847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.371822 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc72bfa4-a197-49da-803b-1336391e40f8-maas-api-openshift-ca" pod "fc72bfa4-a197-49da-803b-1336391e40f8" (UID: "fc72bfa4-a197-49da-803b-1336391e40f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:58.372252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.372231 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc72bfa4-a197-49da-803b-1336391e40f8-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.373800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.373767 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc72bfa4-a197-49da-803b-1336391e40f8-maas-api-xks-ca" pod "fc72bfa4-a197-49da-803b-1336391e40f8" (UID: "fc72bfa4-a197-49da-803b-1336391e40f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:14:58.373910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.373834 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc72bfa4-a197-49da-803b-1336391e40f8-kube-api-access-mcwv2" pod "fc72bfa4-a197-49da-803b-1336391e40f8" (UID: "fc72bfa4-a197-49da-803b-1336391e40f8"). InnerVolumeSpecName "kube-api-access-mcwv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:58.473250 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.473212 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcwv2\" (UniqueName: \"kubernetes.io/projected/fc72bfa4-a197-49da-803b-1336391e40f8-kube-api-access-mcwv2\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.473250 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.473246 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc72bfa4-a197-49da-803b-1336391e40f8-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.588054 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.587911 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf4ba08-f29b-4778-a3ea-b2a0cea035ac" path="/var/lib/kubelet/pods/9bf4ba08-f29b-4778-a3ea-b2a0cea035ac/volumes" Oct 30 02:14:58.658661 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:14:58.658621 5998 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc72bfa4_a197_49da_803b_1336391e40f8.slice\": RecentStats: unable to find data in memory cache]" Oct 30 02:14:58.796106 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:58.796061 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:14:59.051679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:59.051585 5998 generic.go:356] "Generic (PLEG): container finished" podID="563a9420-524b-48cc-9e77-2b6001604841" containerID="2356216a486173038d8e3dc5102885229785d20bddb29cf0ee8de60ee884bc81" exitCode=0 Oct 30 02:14:59.051679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:59.051661 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" event={"ID":"563a9420-524b-48cc-9e77-2b6001604841","Type":"ContainerDied","Data":"2356216a486173038d8e3dc5102885229785d20bddb29cf0ee8de60ee884bc81"} Oct 30 02:14:59.052274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:59.052250 5998 scope.go:122] "RemoveContainer" containerID="2356216a486173038d8e3dc5102885229785d20bddb29cf0ee8de60ee884bc81" Oct 30 02:14:59.053649 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:59.053629 5998 generic.go:356] "Generic (PLEG): container finished" podID="af31090b-f48b-4675-9add-665d0f5d42af" containerID="e9dc92290e7b950f887056916718f686a3923657c26628f386896b13c4723dec" exitCode=0 Oct 30 02:14:59.053747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:59.053655 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" event={"ID":"af31090b-f48b-4675-9add-665d0f5d42af","Type":"ContainerDied","Data":"e9dc92290e7b950f887056916718f686a3923657c26628f386896b13c4723dec"} Oct 30 02:14:59.054064 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:59.054047 5998 scope.go:122] "RemoveContainer" containerID="e9dc92290e7b950f887056916718f686a3923657c26628f386896b13c4723dec" Oct 30 02:14:59.054349 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:59.054313 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:14:59.054484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:59.054410 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:14:59.054484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:59.054469 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:14:59.055869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:59.055842 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:14:59.055954 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:59.055907 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:14:59.055954 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:59.055947 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:14:59.056773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:59.056755 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4" event={"ID":"fc72bfa4-a197-49da-803b-1336391e40f8","Type":"ContainerDied","Data":"6fba2da36a5cfc90677634472ec986b30ba109c8c4d7668e95bc75d33b8abb3a"} Oct 30 02:14:59.056852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:59.056776 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4" Oct 30 02:14:59.056852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:14:59.056796 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fba2da36a5cfc90677634472ec986b30ba109c8c4d7668e95bc75d33b8abb3a" Oct 30 02:15:00.065810 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.065774 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" event={"ID":"af31090b-f48b-4675-9add-665d0f5d42af","Type":"ContainerStarted","Data":"b34f8961d59280e60a050eb9af1aef1a2ed40e9c253ef77f7b4519a0d23dd5eb"} Oct 30 02:15:00.068736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.068700 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" event={"ID":"563a9420-524b-48cc-9e77-2b6001604841","Type":"ContainerStarted","Data":"2606564cb9b531788a30350070a53a0830106b3756f2c6b14874100fe5779950"} Oct 30 02:15:00.169350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.168155 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr"] Oct 30 02:15:00.169350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.169322 5998 cpu_manager.go:414] "containerMap: removing container" podUID="fc72bfa4-a197-49da-803b-1336391e40f8" containerName="revoke-keys" Oct 30 02:15:00.169350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.169350 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc72bfa4-a197-49da-803b-1336391e40f8" containerName="revoke-keys" Oct 30 02:15:00.169662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.169511 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc72bfa4-a197-49da-803b-1336391e40f8" containerName="revoke-keys" Oct 30 02:15:00.173709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.173679 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr"] Oct 30 02:15:00.175889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.174342 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" Oct 30 02:15:00.179247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.179226 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" Oct 30 02:15:00.181171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.180433 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2"] Oct 30 02:15:00.185294 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.185278 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" Oct 30 02:15:00.189819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.189692 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms"] Oct 30 02:15:00.195628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.194119 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:15:00.195628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.194205 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:15:00.195628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.194294 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:15:00.195628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.194685 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s"] Oct 30 02:15:00.196057 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.196031 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" Oct 30 02:15:00.199157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.198232 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw"] Oct 30 02:15:00.199157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.198327 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" Oct 30 02:15:00.263116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.226865 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" Oct 30 02:15:00.288172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.270970 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t"] Oct 30 02:15:00.288172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.282360 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" Oct 30 02:15:00.288172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.287741 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr"] Oct 30 02:15:00.295189 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.295149 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7"] Oct 30 02:15:00.392763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.391360 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a93eb477-297b-4060-a79a-8ba36de67e83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2\" (UID: \"a93eb477-297b-4060-a79a-8ba36de67e83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" Oct 30 02:15:00.392763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.391457 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea963033-f2cb-4e1c-a9ef-7d25c76581eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888775-xz57t\" (UID: \"ea963033-f2cb-4e1c-a9ef-7d25c76581eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" Oct 30 02:15:00.392763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.391557 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndfs8\" (UniqueName: \"kubernetes.io/projected/3212debe-7561-4fa1-bdd9-bd34e88389ea-kube-api-access-ndfs8\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s\" (UID: \"3212debe-7561-4fa1-bdd9-bd34e88389ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" Oct 30 02:15:00.392763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.391599 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk42p\" (UniqueName: \"kubernetes.io/projected/a93eb477-297b-4060-a79a-8ba36de67e83-kube-api-access-pk42p\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2\" (UID: \"a93eb477-297b-4060-a79a-8ba36de67e83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" Oct 30 02:15:00.392763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.391640 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e5a7ed-87b9-4be7-87fe-39944b774797-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms\" (UID: \"54e5a7ed-87b9-4be7-87fe-39944b774797\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" Oct 30 02:15:00.392763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.391676 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v84gw\" (UniqueName: \"kubernetes.io/projected/ab563a5a-c1f8-4e32-aa5d-91effabc8a18-kube-api-access-v84gw\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr\" (UID: \"ab563a5a-c1f8-4e32-aa5d-91effabc8a18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" Oct 30 02:15:00.392763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.391721 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a25d75a-85d3-4f3f-a951-aa39f35481fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw\" (UID: \"0a25d75a-85d3-4f3f-a951-aa39f35481fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" Oct 30 02:15:00.392763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.391787 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm7c9\" (UniqueName: \"kubernetes.io/projected/0a25d75a-85d3-4f3f-a951-aa39f35481fb-kube-api-access-pm7c9\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw\" (UID: \"0a25d75a-85d3-4f3f-a951-aa39f35481fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" Oct 30 02:15:00.392763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.391906 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3212debe-7561-4fa1-bdd9-bd34e88389ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s\" (UID: \"3212debe-7561-4fa1-bdd9-bd34e88389ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" Oct 30 02:15:00.392763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.392013 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab563a5a-c1f8-4e32-aa5d-91effabc8a18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr\" (UID: \"ab563a5a-c1f8-4e32-aa5d-91effabc8a18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" Oct 30 02:15:00.392763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.392042 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr\" (UID: \"3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" Oct 30 02:15:00.392763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.392198 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xfkl\" (UniqueName: \"kubernetes.io/projected/54e5a7ed-87b9-4be7-87fe-39944b774797-kube-api-access-6xfkl\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms\" (UID: \"54e5a7ed-87b9-4be7-87fe-39944b774797\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" Oct 30 02:15:00.392763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.392283 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hcf7\" (UniqueName: \"kubernetes.io/projected/3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4-kube-api-access-6hcf7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr\" (UID: \"3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" Oct 30 02:15:00.392763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.392318 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdbzd\" (UniqueName: \"kubernetes.io/projected/ea963033-f2cb-4e1c-a9ef-7d25c76581eb-kube-api-access-bdbzd\") pod \"maas-api-key-cleanup-29888775-xz57t\" (UID: \"ea963033-f2cb-4e1c-a9ef-7d25c76581eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" Oct 30 02:15:00.392763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.392642 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr"] Oct 30 02:15:00.392763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.392681 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms"] Oct 30 02:15:00.392763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.392771 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" Oct 30 02:15:00.395076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.395056 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb"] Oct 30 02:15:00.477553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.477520 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s"] Oct 30 02:15:00.477796 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.477782 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s"] Oct 30 02:15:00.479456 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.479430 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" Oct 30 02:15:00.482694 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.482674 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb"] Oct 30 02:15:00.482813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.482790 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" Oct 30 02:15:00.487029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.486903 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7"] Oct 30 02:15:00.487029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.486973 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2"] Oct 30 02:15:00.487228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.487062 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" Oct 30 02:15:00.491066 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.491043 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t"] Oct 30 02:15:00.493360 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.493334 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndfs8\" (UniqueName: \"kubernetes.io/projected/3212debe-7561-4fa1-bdd9-bd34e88389ea-kube-api-access-ndfs8\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s\" (UID: \"3212debe-7561-4fa1-bdd9-bd34e88389ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" Oct 30 02:15:00.493547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.493524 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk42p\" (UniqueName: \"kubernetes.io/projected/a93eb477-297b-4060-a79a-8ba36de67e83-kube-api-access-pk42p\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2\" (UID: \"a93eb477-297b-4060-a79a-8ba36de67e83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" Oct 30 02:15:00.493694 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.493677 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e5a7ed-87b9-4be7-87fe-39944b774797-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms\" (UID: \"54e5a7ed-87b9-4be7-87fe-39944b774797\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" Oct 30 02:15:00.493824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.493808 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v84gw\" (UniqueName: \"kubernetes.io/projected/ab563a5a-c1f8-4e32-aa5d-91effabc8a18-kube-api-access-v84gw\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr\" (UID: \"ab563a5a-c1f8-4e32-aa5d-91effabc8a18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" Oct 30 02:15:00.493953 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.493936 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a25d75a-85d3-4f3f-a951-aa39f35481fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw\" (UID: \"0a25d75a-85d3-4f3f-a951-aa39f35481fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" Oct 30 02:15:00.494118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.494083 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm7c9\" (UniqueName: \"kubernetes.io/projected/0a25d75a-85d3-4f3f-a951-aa39f35481fb-kube-api-access-pm7c9\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw\" (UID: \"0a25d75a-85d3-4f3f-a951-aa39f35481fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" Oct 30 02:15:00.494421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.494400 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3212debe-7561-4fa1-bdd9-bd34e88389ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s\" (UID: \"3212debe-7561-4fa1-bdd9-bd34e88389ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" Oct 30 02:15:00.494643 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.494563 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab563a5a-c1f8-4e32-aa5d-91effabc8a18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr\" (UID: \"ab563a5a-c1f8-4e32-aa5d-91effabc8a18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" Oct 30 02:15:00.494783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.494765 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr\" (UID: \"3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" Oct 30 02:15:00.494925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.494909 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xfkl\" (UniqueName: \"kubernetes.io/projected/54e5a7ed-87b9-4be7-87fe-39944b774797-kube-api-access-6xfkl\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms\" (UID: \"54e5a7ed-87b9-4be7-87fe-39944b774797\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" Oct 30 02:15:00.495072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.495057 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hcf7\" (UniqueName: \"kubernetes.io/projected/3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4-kube-api-access-6hcf7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr\" (UID: \"3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" Oct 30 02:15:00.495220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.495204 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdbzd\" (UniqueName: \"kubernetes.io/projected/ea963033-f2cb-4e1c-a9ef-7d25c76581eb-kube-api-access-bdbzd\") pod \"maas-api-key-cleanup-29888775-xz57t\" (UID: \"ea963033-f2cb-4e1c-a9ef-7d25c76581eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" Oct 30 02:15:00.495378 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.495360 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a93eb477-297b-4060-a79a-8ba36de67e83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2\" (UID: \"a93eb477-297b-4060-a79a-8ba36de67e83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" Oct 30 02:15:00.495567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.495549 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea963033-f2cb-4e1c-a9ef-7d25c76581eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888775-xz57t\" (UID: \"ea963033-f2cb-4e1c-a9ef-7d25c76581eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" Oct 30 02:15:00.498144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.496693 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea963033-f2cb-4e1c-a9ef-7d25c76581eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888775-xz57t\" (UID: \"ea963033-f2cb-4e1c-a9ef-7d25c76581eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" Oct 30 02:15:00.498586 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.498560 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3212debe-7561-4fa1-bdd9-bd34e88389ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s\" (UID: \"3212debe-7561-4fa1-bdd9-bd34e88389ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" Oct 30 02:15:00.499122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.499100 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab563a5a-c1f8-4e32-aa5d-91effabc8a18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr\" (UID: \"ab563a5a-c1f8-4e32-aa5d-91effabc8a18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" Oct 30 02:15:00.499654 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.499625 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e5a7ed-87b9-4be7-87fe-39944b774797-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms\" (UID: \"54e5a7ed-87b9-4be7-87fe-39944b774797\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" Oct 30 02:15:00.499855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.499834 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr\" (UID: \"3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" Oct 30 02:15:00.554117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.501143 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a93eb477-297b-4060-a79a-8ba36de67e83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2\" (UID: \"a93eb477-297b-4060-a79a-8ba36de67e83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" Oct 30 02:15:00.577687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.520259 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb"] Oct 30 02:15:00.581963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.574067 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a25d75a-85d3-4f3f-a951-aa39f35481fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw\" (UID: \"0a25d75a-85d3-4f3f-a951-aa39f35481fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" Oct 30 02:15:00.581963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.575636 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk42p\" (UniqueName: \"kubernetes.io/projected/a93eb477-297b-4060-a79a-8ba36de67e83-kube-api-access-pk42p\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2\" (UID: \"a93eb477-297b-4060-a79a-8ba36de67e83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" Oct 30 02:15:00.582741 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.582711 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 02:15:00.583546 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.583458 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" podUID="af31090b-f48b-4675-9add-665d0f5d42af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 02:15:00.589853 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.589758 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hcf7\" (UniqueName: \"kubernetes.io/projected/3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4-kube-api-access-6hcf7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr\" (UID: \"3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" Oct 30 02:15:00.590574 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.590498 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xfkl\" (UniqueName: \"kubernetes.io/projected/54e5a7ed-87b9-4be7-87fe-39944b774797-kube-api-access-6xfkl\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms\" (UID: \"54e5a7ed-87b9-4be7-87fe-39944b774797\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" Oct 30 02:15:00.591817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.591642 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm7c9\" (UniqueName: \"kubernetes.io/projected/0a25d75a-85d3-4f3f-a951-aa39f35481fb-kube-api-access-pm7c9\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw\" (UID: \"0a25d75a-85d3-4f3f-a951-aa39f35481fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" Oct 30 02:15:00.597140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.595736 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndfs8\" (UniqueName: \"kubernetes.io/projected/3212debe-7561-4fa1-bdd9-bd34e88389ea-kube-api-access-ndfs8\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s\" (UID: \"3212debe-7561-4fa1-bdd9-bd34e88389ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" Oct 30 02:15:00.597140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.596047 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v84gw\" (UniqueName: \"kubernetes.io/projected/ab563a5a-c1f8-4e32-aa5d-91effabc8a18-kube-api-access-v84gw\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr\" (UID: \"ab563a5a-c1f8-4e32-aa5d-91effabc8a18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" Oct 30 02:15:00.598361 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.597811 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/287b0a33-1dc7-4284-8c93-7af661cf9d37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7\" (UID: \"287b0a33-1dc7-4284-8c93-7af661cf9d37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" Oct 30 02:15:00.598361 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.597859 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdbzd\" (UniqueName: \"kubernetes.io/projected/ea963033-f2cb-4e1c-a9ef-7d25c76581eb-kube-api-access-bdbzd\") pod \"maas-api-key-cleanup-29888775-xz57t\" (UID: \"ea963033-f2cb-4e1c-a9ef-7d25c76581eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" Oct 30 02:15:00.598361 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.597871 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffvj5\" (UniqueName: \"kubernetes.io/projected/287b0a33-1dc7-4284-8c93-7af661cf9d37-kube-api-access-ffvj5\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7\" (UID: \"287b0a33-1dc7-4284-8c93-7af661cf9d37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" Oct 30 02:15:00.598361 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.598349 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf51a9f7-5514-42fd-9122-75d1495cde92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb\" (UID: \"bf51a9f7-5514-42fd-9122-75d1495cde92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" Oct 30 02:15:00.598671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.598426 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d97zr\" (UniqueName: \"kubernetes.io/projected/bf51a9f7-5514-42fd-9122-75d1495cde92-kube-api-access-d97zr\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb\" (UID: \"bf51a9f7-5514-42fd-9122-75d1495cde92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" Oct 30 02:15:00.598671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.598471 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d6d3ad-5fa9-4aca-9803-a981213c42af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb\" (UID: \"88d6d3ad-5fa9-4aca-9803-a981213c42af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" Oct 30 02:15:00.598671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.598495 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxnbh\" (UniqueName: \"kubernetes.io/projected/f70185f3-a978-4c11-ab66-903cfcba41b8-kube-api-access-nxnbh\") pod \"maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s\" (UID: \"f70185f3-a978-4c11-ab66-903cfcba41b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" Oct 30 02:15:00.598671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.598576 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhqh8\" (UniqueName: \"kubernetes.io/projected/88d6d3ad-5fa9-4aca-9803-a981213c42af-kube-api-access-mhqh8\") pod \"maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb\" (UID: \"88d6d3ad-5fa9-4aca-9803-a981213c42af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" Oct 30 02:15:00.598671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.598660 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70185f3-a978-4c11-ab66-903cfcba41b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s\" (UID: \"f70185f3-a978-4c11-ab66-903cfcba41b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" Oct 30 02:15:00.622398 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.611585 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:15:00.622398 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.611684 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw"] Oct 30 02:15:00.622398 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.611717 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc"] Oct 30 02:15:00.670568 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.670306 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s"] Oct 30 02:15:00.670568 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.670400 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" Oct 30 02:15:00.670918 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.670885 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:00.672161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.672136 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" Oct 30 02:15:00.672489 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.672473 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf"] Oct 30 02:15:00.677042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.676022 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" Oct 30 02:15:00.686643 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.686601 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf"] Oct 30 02:15:00.687814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.687791 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" Oct 30 02:15:00.689531 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.689506 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc"] Oct 30 02:15:00.692019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.692000 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" Oct 30 02:15:00.713609 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.710637 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" Oct 30 02:15:00.737581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.736001 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vqct\" (UniqueName: \"kubernetes.io/projected/30276c96-e1d6-4759-9368-51b819812537-kube-api-access-9vqct\") pod \"maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf\" (UID: \"30276c96-e1d6-4759-9368-51b819812537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" Oct 30 02:15:00.737581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.736065 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/287b0a33-1dc7-4284-8c93-7af661cf9d37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7\" (UID: \"287b0a33-1dc7-4284-8c93-7af661cf9d37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" Oct 30 02:15:00.737581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.736145 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a692aea-70f7-4f61-adc6-bba42c9cc1bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc\" (UID: \"7a692aea-70f7-4f61-adc6-bba42c9cc1bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" Oct 30 02:15:00.737581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.736175 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30276c96-e1d6-4759-9368-51b819812537-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf\" (UID: \"30276c96-e1d6-4759-9368-51b819812537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" Oct 30 02:15:00.737581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.736217 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffvj5\" (UniqueName: \"kubernetes.io/projected/287b0a33-1dc7-4284-8c93-7af661cf9d37-kube-api-access-ffvj5\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7\" (UID: \"287b0a33-1dc7-4284-8c93-7af661cf9d37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" Oct 30 02:15:00.737581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.736526 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf51a9f7-5514-42fd-9122-75d1495cde92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb\" (UID: \"bf51a9f7-5514-42fd-9122-75d1495cde92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" Oct 30 02:15:00.737581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.736580 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rr2m\" (UniqueName: \"kubernetes.io/projected/7a692aea-70f7-4f61-adc6-bba42c9cc1bd-kube-api-access-8rr2m\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc\" (UID: \"7a692aea-70f7-4f61-adc6-bba42c9cc1bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" Oct 30 02:15:00.737581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.736646 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d97zr\" (UniqueName: \"kubernetes.io/projected/bf51a9f7-5514-42fd-9122-75d1495cde92-kube-api-access-d97zr\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb\" (UID: \"bf51a9f7-5514-42fd-9122-75d1495cde92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" Oct 30 02:15:00.737581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.736688 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d6d3ad-5fa9-4aca-9803-a981213c42af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb\" (UID: \"88d6d3ad-5fa9-4aca-9803-a981213c42af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" Oct 30 02:15:00.737581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.736716 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnbh\" (UniqueName: \"kubernetes.io/projected/f70185f3-a978-4c11-ab66-903cfcba41b8-kube-api-access-nxnbh\") pod \"maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s\" (UID: \"f70185f3-a978-4c11-ab66-903cfcba41b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" Oct 30 02:15:00.737581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.736813 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhqh8\" (UniqueName: \"kubernetes.io/projected/88d6d3ad-5fa9-4aca-9803-a981213c42af-kube-api-access-mhqh8\") pod \"maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb\" (UID: \"88d6d3ad-5fa9-4aca-9803-a981213c42af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" Oct 30 02:15:00.737581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.736848 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70185f3-a978-4c11-ab66-903cfcba41b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s\" (UID: \"f70185f3-a978-4c11-ab66-903cfcba41b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" Oct 30 02:15:00.739117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.738645 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf51a9f7-5514-42fd-9122-75d1495cde92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb\" (UID: \"bf51a9f7-5514-42fd-9122-75d1495cde92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" Oct 30 02:15:00.763143 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.739610 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/287b0a33-1dc7-4284-8c93-7af661cf9d37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7\" (UID: \"287b0a33-1dc7-4284-8c93-7af661cf9d37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" Oct 30 02:15:00.787123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.783818 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70185f3-a978-4c11-ab66-903cfcba41b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s\" (UID: \"f70185f3-a978-4c11-ab66-903cfcba41b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" Oct 30 02:15:00.791115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.787936 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d6d3ad-5fa9-4aca-9803-a981213c42af-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb\" (UID: \"88d6d3ad-5fa9-4aca-9803-a981213c42af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" Oct 30 02:15:00.794534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.794471 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" Oct 30 02:15:00.816443 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.809499 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb"] Oct 30 02:15:00.830122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.820804 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffvj5\" (UniqueName: \"kubernetes.io/projected/287b0a33-1dc7-4284-8c93-7af661cf9d37-kube-api-access-ffvj5\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7\" (UID: \"287b0a33-1dc7-4284-8c93-7af661cf9d37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" Oct 30 02:15:00.861202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.840256 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rr2m\" (UniqueName: \"kubernetes.io/projected/7a692aea-70f7-4f61-adc6-bba42c9cc1bd-kube-api-access-8rr2m\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc\" (UID: \"7a692aea-70f7-4f61-adc6-bba42c9cc1bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" Oct 30 02:15:00.861202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.840487 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vqct\" (UniqueName: \"kubernetes.io/projected/30276c96-e1d6-4759-9368-51b819812537-kube-api-access-9vqct\") pod \"maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf\" (UID: \"30276c96-e1d6-4759-9368-51b819812537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" Oct 30 02:15:00.861202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.840548 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a692aea-70f7-4f61-adc6-bba42c9cc1bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc\" (UID: \"7a692aea-70f7-4f61-adc6-bba42c9cc1bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" Oct 30 02:15:00.861202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.840574 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30276c96-e1d6-4759-9368-51b819812537-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf\" (UID: \"30276c96-e1d6-4759-9368-51b819812537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" Oct 30 02:15:00.861202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.841578 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30276c96-e1d6-4759-9368-51b819812537-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf\" (UID: \"30276c96-e1d6-4759-9368-51b819812537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" Oct 30 02:15:00.861202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.860257 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a692aea-70f7-4f61-adc6-bba42c9cc1bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc\" (UID: \"7a692aea-70f7-4f61-adc6-bba42c9cc1bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" Oct 30 02:15:00.861202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.860692 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" Oct 30 02:15:00.865080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.865044 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d97zr\" (UniqueName: \"kubernetes.io/projected/bf51a9f7-5514-42fd-9122-75d1495cde92-kube-api-access-d97zr\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb\" (UID: \"bf51a9f7-5514-42fd-9122-75d1495cde92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" Oct 30 02:15:00.875818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.875784 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxnbh\" (UniqueName: \"kubernetes.io/projected/f70185f3-a978-4c11-ab66-903cfcba41b8-kube-api-access-nxnbh\") pod \"maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s\" (UID: \"f70185f3-a978-4c11-ab66-903cfcba41b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" Oct 30 02:15:00.877362 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.877317 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhqh8\" (UniqueName: \"kubernetes.io/projected/88d6d3ad-5fa9-4aca-9803-a981213c42af-kube-api-access-mhqh8\") pod \"maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb\" (UID: \"88d6d3ad-5fa9-4aca-9803-a981213c42af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" Oct 30 02:15:00.878653 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.878628 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" Oct 30 02:15:00.881183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.881154 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vqct\" (UniqueName: \"kubernetes.io/projected/30276c96-e1d6-4759-9368-51b819812537-kube-api-access-9vqct\") pod \"maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf\" (UID: \"30276c96-e1d6-4759-9368-51b819812537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" Oct 30 02:15:00.882543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.882427 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" Oct 30 02:15:00.885415 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.884886 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rr2m\" (UniqueName: \"kubernetes.io/projected/7a692aea-70f7-4f61-adc6-bba42c9cc1bd-kube-api-access-8rr2m\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc\" (UID: \"7a692aea-70f7-4f61-adc6-bba42c9cc1bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" Oct 30 02:15:00.945333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.934880 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" Oct 30 02:15:00.945333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.936681 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:15:00.945333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.937013 5998 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 02:15:00.945333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.937055 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" podUID="563a9420-524b-48cc-9e77-2b6001604841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 02:15:00.976455 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.973341 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" Oct 30 02:15:00.998383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:00.997435 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" Oct 30 02:15:01.098551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:01.098476 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" Oct 30 02:15:01.164277 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:01.164038 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" Oct 30 02:15:01.462817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:01.461140 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw"] Oct 30 02:15:01.574132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:01.569947 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 02:15:01.574132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:01.570009 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" podUID="af31090b-f48b-4675-9add-665d0f5d42af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 02:15:01.678356 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:15:01.678325 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a25d75a_85d3_4f3f_a951_aa39f35481fb.slice/crio-f26c0fe08a8ee15db61afc96f4e73f20764a61717f1a079ce14e3231ad632d2a WatchSource:0}: Error finding container f26c0fe08a8ee15db61afc96f4e73f20764a61717f1a079ce14e3231ad632d2a: Status 404 returned error can't find the container with id f26c0fe08a8ee15db61afc96f4e73f20764a61717f1a079ce14e3231ad632d2a Oct 30 02:15:01.960703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:01.959581 5998 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 02:15:01.960703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:01.959638 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" podUID="563a9420-524b-48cc-9e77-2b6001604841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 02:15:02.343375 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:02.343344 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t"] Oct 30 02:15:02.454293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:02.452875 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2"] Oct 30 02:15:02.470827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:02.470616 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" event={"ID":"0a25d75a-85d3-4f3f-a951-aa39f35481fb","Type":"ContainerStarted","Data":"f26c0fe08a8ee15db61afc96f4e73f20764a61717f1a079ce14e3231ad632d2a"} Oct 30 02:15:02.479836 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:02.479774 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:02.679336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:02.678845 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:02.679336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:02.679064 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 02:15:02.683148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:02.679259 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" podUID="af31090b-f48b-4675-9add-665d0f5d42af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 02:15:02.778600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:02.776769 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph"] Oct 30 02:15:02.778600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:02.776958 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" podUID="1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" containerName="istio-proxy" containerID="cri-o://cb0b55c49f2ada5676d0d7b8de4dcc9cf8ff84c1ac3da34f992344ce5e0cf39b" gracePeriod=30 Oct 30 02:15:02.960118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:02.949861 5998 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 02:15:02.960118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:02.949920 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" podUID="563a9420-524b-48cc-9e77-2b6001604841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 02:15:02.973454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:02.972920 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr"] Oct 30 02:15:02.979440 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:02.979340 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb"] Oct 30 02:15:02.998507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:02.992675 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms"] Oct 30 02:15:02.998507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:02.997793 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s"] Oct 30 02:15:03.095906 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:15:03.095349 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70185f3_a978_4c11_ab66_903cfcba41b8.slice/crio-961ee16e7f9d6423f25de0ef68c0391edc7bd037662169ee712a28e935ded8de WatchSource:0}: Error finding container 961ee16e7f9d6423f25de0ef68c0391edc7bd037662169ee712a28e935ded8de: Status 404 returned error can't find the container with id 961ee16e7f9d6423f25de0ef68c0391edc7bd037662169ee712a28e935ded8de Oct 30 02:15:03.191284 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.187046 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf"] Oct 30 02:15:03.191284 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.190481 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb"] Oct 30 02:15:03.194220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.194077 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7"] Oct 30 02:15:03.197351 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:15:03.197317 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3212debe_7561_4fa1_bdd9_bd34e88389ea.slice/crio-72d7ca6eb883a0e7c75f6b61290fff9c651caf99a2568eb67972906eb8e420cf WatchSource:0}: Error finding container 72d7ca6eb883a0e7c75f6b61290fff9c651caf99a2568eb67972906eb8e420cf: Status 404 returned error can't find the container with id 72d7ca6eb883a0e7c75f6b61290fff9c651caf99a2568eb67972906eb8e420cf Oct 30 02:15:03.221501 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:15:03.199506 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d6d3ad_5fa9_4aca_9803_a981213c42af.slice/crio-d57bfdfe84268db398581bf0449b96e5d67f6a2ff91927b417c0be2a1c3c102f WatchSource:0}: Error finding container d57bfdfe84268db398581bf0449b96e5d67f6a2ff91927b417c0be2a1c3c102f: Status 404 returned error can't find the container with id d57bfdfe84268db398581bf0449b96e5d67f6a2ff91927b417c0be2a1c3c102f Oct 30 02:15:03.279591 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.223651 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr"] Oct 30 02:15:03.279591 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.278947 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc"] Oct 30 02:15:03.286846 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.285249 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s"] Oct 30 02:15:03.487240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.487216 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" event={"ID":"3212debe-7561-4fa1-bdd9-bd34e88389ea","Type":"ContainerStarted","Data":"72d7ca6eb883a0e7c75f6b61290fff9c651caf99a2568eb67972906eb8e420cf"} Oct 30 02:15:03.488704 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.488678 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" event={"ID":"88d6d3ad-5fa9-4aca-9803-a981213c42af","Type":"ContainerStarted","Data":"d57bfdfe84268db398581bf0449b96e5d67f6a2ff91927b417c0be2a1c3c102f"} Oct 30 02:15:03.490057 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.490039 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" event={"ID":"54e5a7ed-87b9-4be7-87fe-39944b774797","Type":"ContainerStarted","Data":"f41d0da0a098716b80a2f3410ce55dc1bb9c88281b3d6abea58538bc4d06883b"} Oct 30 02:15:03.491763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.491570 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" event={"ID":"bf51a9f7-5514-42fd-9122-75d1495cde92","Type":"ContainerStarted","Data":"d3c1fa0d604936191a3e3cd80a4c5c2b771000961ac9ec521c776b2dc874ed12"} Oct 30 02:15:03.493099 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.493068 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" event={"ID":"30276c96-e1d6-4759-9368-51b819812537","Type":"ContainerStarted","Data":"474349e3c9a122ab19726eb0af82ebb327a0f91bef2f309bb890483afc543fe6"} Oct 30 02:15:03.494353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.494333 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" event={"ID":"ab563a5a-c1f8-4e32-aa5d-91effabc8a18","Type":"ContainerStarted","Data":"48042c6da38079479657b986d12da2433234507956ae08a8a48842702359f8b8"} Oct 30 02:15:03.498615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.498585 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" event={"ID":"287b0a33-1dc7-4284-8c93-7af661cf9d37","Type":"ContainerStarted","Data":"0ad7c3ae07375925ab6fc87ac48919c7a915d70762e9aa1404c12c1368cc3b2c"} Oct 30 02:15:03.569891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.569858 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 02:15:03.570051 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.569917 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" podUID="af31090b-f48b-4675-9add-665d0f5d42af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 02:15:03.570789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.570756 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" event={"ID":"3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4","Type":"ContainerStarted","Data":"11581371b1bcbf349c6bf58622113f714c8c7352444ae6de58ab3c09ffa0793d"} Oct 30 02:15:03.573957 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.573919 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" event={"ID":"a93eb477-297b-4060-a79a-8ba36de67e83","Type":"ContainerStarted","Data":"76201069ee9a86ecb54c6711fa6c3f888ab77b42ecf9cdeec5fcbd63473a4f27"} Oct 30 02:15:03.574196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.574172 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" event={"ID":"a93eb477-297b-4060-a79a-8ba36de67e83","Type":"ContainerStarted","Data":"dfbb2c649e273688da3094f1c51c2df1e76d7757f7e36915b986958448e27afa"} Oct 30 02:15:03.578601 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.578569 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" event={"ID":"ea963033-f2cb-4e1c-a9ef-7d25c76581eb","Type":"ContainerStarted","Data":"aea588db8423e78647f018db2b24bfec6520f320db5fd868db8ffdc13463adc2"} Oct 30 02:15:03.579725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.579694 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" event={"ID":"ea963033-f2cb-4e1c-a9ef-7d25c76581eb","Type":"ContainerStarted","Data":"2351acb341582aef755653526338c1e2cdbc2f4e157f96d67a0b15a7039199e2"} Oct 30 02:15:03.586934 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.586892 5998 generic.go:356] "Generic (PLEG): container finished" podID="0a25d75a-85d3-4f3f-a951-aa39f35481fb" containerID="553f4e03c3348d7184fb99682c2a6f52d454490a131415952b5330dd991e4ab6" exitCode=60 Oct 30 02:15:03.587064 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.586981 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" event={"ID":"0a25d75a-85d3-4f3f-a951-aa39f35481fb","Type":"ContainerDied","Data":"553f4e03c3348d7184fb99682c2a6f52d454490a131415952b5330dd991e4ab6"} Oct 30 02:15:03.587397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.587340 5998 scope.go:122] "RemoveContainer" containerID="553f4e03c3348d7184fb99682c2a6f52d454490a131415952b5330dd991e4ab6" Oct 30 02:15:03.592953 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.592932 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" event={"ID":"7a692aea-70f7-4f61-adc6-bba42c9cc1bd","Type":"ContainerStarted","Data":"2bd3a609f230fc6f3aa7a45382e1becadeec9dfe00b22dee6764d1e7b34fd273"} Oct 30 02:15:03.595462 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.595440 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" event={"ID":"f70185f3-a978-4c11-ab66-903cfcba41b8","Type":"ContainerStarted","Data":"961ee16e7f9d6423f25de0ef68c0391edc7bd037662169ee712a28e935ded8de"} Oct 30 02:15:03.675771 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.675701 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" podStartSLOduration=3.675677763 podStartE2EDuration="3.675677763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:03.660303822 +0000 UTC m=+2733.239094895" watchObservedRunningTime="2026-10-30 02:15:03.675677763 +0000 UTC m=+2733.254468844" Oct 30 02:15:03.780911 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.777587 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" podStartSLOduration=3.777555617 podStartE2EDuration="3.777555617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:03.773639951 +0000 UTC m=+2733.352431032" watchObservedRunningTime="2026-10-30 02:15:03.777555617 +0000 UTC m=+2733.356346699" Oct 30 02:15:03.937678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.937533 5998 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 02:15:03.937678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.937600 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" podUID="563a9420-524b-48cc-9e77-2b6001604841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 02:15:03.964538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:03.964457 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:04.183925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:04.183892 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:04.508173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:04.491226 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:04.588155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:04.584786 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 02:15:04.588155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:04.584847 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" podUID="af31090b-f48b-4675-9add-665d0f5d42af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 02:15:04.666315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:04.630046 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:05.046696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:04.979835 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:05.046696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:04.985264 5998 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 02:15:05.046696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:04.985324 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" podUID="563a9420-524b-48cc-9e77-2b6001604841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 02:15:05.155124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.144658 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7"] Oct 30 02:15:05.176113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.168736 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" podStartSLOduration=5.168714102 podStartE2EDuration="5.168714102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:05.16546955 +0000 UTC m=+2734.744260636" watchObservedRunningTime="2026-10-30 02:15:05.168714102 +0000 UTC m=+2734.747505186" Oct 30 02:15:05.194212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.194049 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7" Oct 30 02:15:05.195422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.193953 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" event={"ID":"287b0a33-1dc7-4284-8c93-7af661cf9d37","Type":"ContainerStarted","Data":"307ee30e30037fc5821854104b6ffcdbf81ac61734089387493384debcd82eb6"} Oct 30 02:15:05.195621 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.195608 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7"] Oct 30 02:15:05.215409 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.213899 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:15:05.296229 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.294593 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5vsw\" (UniqueName: \"kubernetes.io/projected/89f01e25-ada7-489b-966d-823f8bc623a1-kube-api-access-k5vsw\") pod \"maas-api-revoke-sub-0114773b-jg4s7\" (UID: \"89f01e25-ada7-489b-966d-823f8bc623a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7" Oct 30 02:15:05.296229 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.294770 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f01e25-ada7-489b-966d-823f8bc623a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0114773b-jg4s7\" (UID: \"89f01e25-ada7-489b-966d-823f8bc623a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7" Oct 30 02:15:05.296229 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.294815 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f01e25-ada7-489b-966d-823f8bc623a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0114773b-jg4s7\" (UID: \"89f01e25-ada7-489b-966d-823f8bc623a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7" Oct 30 02:15:05.315570 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.315521 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" event={"ID":"3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4","Type":"ContainerStarted","Data":"fa8291a4d07f31b776be6da55ecdc3cb0f37bb0aa1e2c23f180b0449fd31c9c7"} Oct 30 02:15:05.384492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.383840 5998 generic.go:356] "Generic (PLEG): container finished" podID="a93eb477-297b-4060-a79a-8ba36de67e83" containerID="76201069ee9a86ecb54c6711fa6c3f888ab77b42ecf9cdeec5fcbd63473a4f27" exitCode=60 Oct 30 02:15:05.384492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.383966 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" event={"ID":"a93eb477-297b-4060-a79a-8ba36de67e83","Type":"ContainerDied","Data":"76201069ee9a86ecb54c6711fa6c3f888ab77b42ecf9cdeec5fcbd63473a4f27"} Oct 30 02:15:05.385458 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.384999 5998 scope.go:122] "RemoveContainer" containerID="76201069ee9a86ecb54c6711fa6c3f888ab77b42ecf9cdeec5fcbd63473a4f27" Oct 30 02:15:05.399706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.399666 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f01e25-ada7-489b-966d-823f8bc623a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0114773b-jg4s7\" (UID: \"89f01e25-ada7-489b-966d-823f8bc623a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7" Oct 30 02:15:05.399967 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.399948 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f01e25-ada7-489b-966d-823f8bc623a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0114773b-jg4s7\" (UID: \"89f01e25-ada7-489b-966d-823f8bc623a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7" Oct 30 02:15:05.400277 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.400258 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vsw\" (UniqueName: \"kubernetes.io/projected/89f01e25-ada7-489b-966d-823f8bc623a1-kube-api-access-k5vsw\") pod \"maas-api-revoke-sub-0114773b-jg4s7\" (UID: \"89f01e25-ada7-489b-966d-823f8bc623a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7" Oct 30 02:15:05.427779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.427737 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f01e25-ada7-489b-966d-823f8bc623a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0114773b-jg4s7\" (UID: \"89f01e25-ada7-489b-966d-823f8bc623a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7" Oct 30 02:15:05.429080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.428842 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f01e25-ada7-489b-966d-823f8bc623a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0114773b-jg4s7\" (UID: \"89f01e25-ada7-489b-966d-823f8bc623a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7" Oct 30 02:15:05.477961 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.472904 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vsw\" (UniqueName: \"kubernetes.io/projected/89f01e25-ada7-489b-966d-823f8bc623a1-kube-api-access-k5vsw\") pod \"maas-api-revoke-sub-0114773b-jg4s7\" (UID: \"89f01e25-ada7-489b-966d-823f8bc623a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7" Oct 30 02:15:05.478469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.476634 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" podStartSLOduration=5.476616158 podStartE2EDuration="5.476616158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:05.39750573 +0000 UTC m=+2734.976296812" watchObservedRunningTime="2026-10-30 02:15:05.476616158 +0000 UTC m=+2735.055407241" Oct 30 02:15:05.479484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.479456 5998 generic.go:356] "Generic (PLEG): container finished" podID="ea963033-f2cb-4e1c-a9ef-7d25c76581eb" containerID="aea588db8423e78647f018db2b24bfec6520f320db5fd868db8ffdc13463adc2" exitCode=60 Oct 30 02:15:05.479609 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.479587 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" event={"ID":"ea963033-f2cb-4e1c-a9ef-7d25c76581eb","Type":"ContainerDied","Data":"aea588db8423e78647f018db2b24bfec6520f320db5fd868db8ffdc13463adc2"} Oct 30 02:15:05.480196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.480168 5998 scope.go:122] "RemoveContainer" containerID="aea588db8423e78647f018db2b24bfec6520f320db5fd868db8ffdc13463adc2" Oct 30 02:15:05.484010 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.483973 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" event={"ID":"0a25d75a-85d3-4f3f-a951-aa39f35481fb","Type":"ContainerStarted","Data":"9e38753c060338b49e00edd4b78e366f8114eb7a2dea6f519ea7ad6949496044"} Oct 30 02:15:05.484504 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.484481 5998 scope.go:122] "RemoveContainer" containerID="9e38753c060338b49e00edd4b78e366f8114eb7a2dea6f519ea7ad6949496044" Oct 30 02:15:05.485129 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:05.484874 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw_odh-ai-gateway-infra(0a25d75a-85d3-4f3f-a951-aa39f35481fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" podUID="0a25d75a-85d3-4f3f-a951-aa39f35481fb" Oct 30 02:15:05.491998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.487215 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" event={"ID":"7a692aea-70f7-4f61-adc6-bba42c9cc1bd","Type":"ContainerStarted","Data":"97d51b0a0f9445e428fb78d7b1a39c6a9c60f6fe4d593a4e60632a8f864fa5e1"} Oct 30 02:15:05.491998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.488822 5998 scope.go:122] "RemoveContainer" containerID="97d51b0a0f9445e428fb78d7b1a39c6a9c60f6fe4d593a4e60632a8f864fa5e1" Oct 30 02:15:05.493126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.493074 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" event={"ID":"f70185f3-a978-4c11-ab66-903cfcba41b8","Type":"ContainerStarted","Data":"8aef59d64f6cca76cf005bc8fdeaf7a8aee19e6213b2d44be7799c9235feaf13"} Oct 30 02:15:05.494045 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.494023 5998 scope.go:122] "RemoveContainer" containerID="8aef59d64f6cca76cf005bc8fdeaf7a8aee19e6213b2d44be7799c9235feaf13" Oct 30 02:15:05.495495 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.495471 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" event={"ID":"3212debe-7561-4fa1-bdd9-bd34e88389ea","Type":"ContainerStarted","Data":"0213e37e45852378455d942e474bd072504d6b685f87e004cc473529558fec7c"} Oct 30 02:15:05.495618 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.495593 5998 scope.go:122] "RemoveContainer" containerID="0213e37e45852378455d942e474bd072504d6b685f87e004cc473529558fec7c" Oct 30 02:15:05.497490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.497468 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" event={"ID":"88d6d3ad-5fa9-4aca-9803-a981213c42af","Type":"ContainerStarted","Data":"a2eff0a6be011ac428c64992b7028eda977fe11eba491779da7f96823fb3b2cc"} Oct 30 02:15:05.498022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.497907 5998 scope.go:122] "RemoveContainer" containerID="a2eff0a6be011ac428c64992b7028eda977fe11eba491779da7f96823fb3b2cc" Oct 30 02:15:05.499823 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.499806 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" event={"ID":"54e5a7ed-87b9-4be7-87fe-39944b774797","Type":"ContainerStarted","Data":"3d4a7bed67059fb9d2369d99edb7e8c618a55b9e17f21874f254c830e937f263"} Oct 30 02:15:05.500035 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.500021 5998 scope.go:122] "RemoveContainer" containerID="3d4a7bed67059fb9d2369d99edb7e8c618a55b9e17f21874f254c830e937f263" Oct 30 02:15:05.572466 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.572430 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 02:15:05.572895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.572736 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" podUID="af31090b-f48b-4675-9add-665d0f5d42af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 02:15:05.577446 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.577417 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" event={"ID":"bf51a9f7-5514-42fd-9122-75d1495cde92","Type":"ContainerStarted","Data":"535496fe2a36e44071e00874cbe975826ec6b6948a411ee524982a601da10d45"} Oct 30 02:15:05.577806 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.577779 5998 scope.go:122] "RemoveContainer" containerID="535496fe2a36e44071e00874cbe975826ec6b6948a411ee524982a601da10d45" Oct 30 02:15:05.583997 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.583969 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" event={"ID":"30276c96-e1d6-4759-9368-51b819812537","Type":"ContainerStarted","Data":"56e1fad7611bfd4b3fad3a47a99feddef7d7811b92266ff989040c2ac6173e2d"} Oct 30 02:15:05.584434 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.584416 5998 scope.go:122] "RemoveContainer" containerID="56e1fad7611bfd4b3fad3a47a99feddef7d7811b92266ff989040c2ac6173e2d" Oct 30 02:15:05.589721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.589321 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" event={"ID":"ab563a5a-c1f8-4e32-aa5d-91effabc8a18","Type":"ContainerStarted","Data":"6d7e2d396f865d312c14b29e06ceb9b7eb751c7cb4bb05b0ba0eac9f5bff46c6"} Oct 30 02:15:05.589721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.589625 5998 scope.go:122] "RemoveContainer" containerID="6d7e2d396f865d312c14b29e06ceb9b7eb751c7cb4bb05b0ba0eac9f5bff46c6" Oct 30 02:15:05.659119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:05.658343 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7" Oct 30 02:15:06.022074 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.022003 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:06.374376 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:15:06.374329 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f01e25_ada7_489b_966d_823f8bc623a1.slice/crio-1ad77659559040c1ce69d7006db85a594b73275b33541a63b43280390358dd06 WatchSource:0}: Error finding container 1ad77659559040c1ce69d7006db85a594b73275b33541a63b43280390358dd06: Status 404 returned error can't find the container with id 1ad77659559040c1ce69d7006db85a594b73275b33541a63b43280390358dd06 Oct 30 02:15:06.375767 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.375738 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7"] Oct 30 02:15:06.414307 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.414277 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:15:06.415898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.415864 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:15:06.422154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.421786 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:15:06.575052 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.575026 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:15:06.575762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.575744 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:15:06.576818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.576795 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:15:06.623973 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.623888 5998 generic.go:356] "Generic (PLEG): container finished" podID="287b0a33-1dc7-4284-8c93-7af661cf9d37" containerID="307ee30e30037fc5821854104b6ffcdbf81ac61734089387493384debcd82eb6" exitCode=60 Oct 30 02:15:06.624186 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.623972 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" event={"ID":"287b0a33-1dc7-4284-8c93-7af661cf9d37","Type":"ContainerDied","Data":"307ee30e30037fc5821854104b6ffcdbf81ac61734089387493384debcd82eb6"} Oct 30 02:15:06.624348 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.624329 5998 scope.go:122] "RemoveContainer" containerID="307ee30e30037fc5821854104b6ffcdbf81ac61734089387493384debcd82eb6" Oct 30 02:15:06.627138 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.627080 5998 generic.go:356] "Generic (PLEG): container finished" podID="3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4" containerID="fa8291a4d07f31b776be6da55ecdc3cb0f37bb0aa1e2c23f180b0449fd31c9c7" exitCode=60 Oct 30 02:15:06.627270 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.627251 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" event={"ID":"3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4","Type":"ContainerDied","Data":"fa8291a4d07f31b776be6da55ecdc3cb0f37bb0aa1e2c23f180b0449fd31c9c7"} Oct 30 02:15:06.627586 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.627567 5998 scope.go:122] "RemoveContainer" containerID="fa8291a4d07f31b776be6da55ecdc3cb0f37bb0aa1e2c23f180b0449fd31c9c7" Oct 30 02:15:06.634980 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.634902 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" event={"ID":"a93eb477-297b-4060-a79a-8ba36de67e83","Type":"ContainerStarted","Data":"5e746fcfb42b8f50700442a24860f6ca573d5e0ac79beea3d9a47505707c7a45"} Oct 30 02:15:06.638030 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.638004 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" event={"ID":"ea963033-f2cb-4e1c-a9ef-7d25c76581eb","Type":"ContainerStarted","Data":"4bd0d577b2c02c1076354dca6905804599b930fef79bcfce3235f7aaea5b4187"} Oct 30 02:15:06.642614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.642587 5998 generic.go:356] "Generic (PLEG): container finished" podID="0a25d75a-85d3-4f3f-a951-aa39f35481fb" containerID="9e38753c060338b49e00edd4b78e366f8114eb7a2dea6f519ea7ad6949496044" exitCode=60 Oct 30 02:15:06.642936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.642879 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" event={"ID":"0a25d75a-85d3-4f3f-a951-aa39f35481fb","Type":"ContainerDied","Data":"9e38753c060338b49e00edd4b78e366f8114eb7a2dea6f519ea7ad6949496044"} Oct 30 02:15:06.643033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.642953 5998 scope.go:122] "RemoveContainer" containerID="553f4e03c3348d7184fb99682c2a6f52d454490a131415952b5330dd991e4ab6" Oct 30 02:15:06.643033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.642992 5998 scope.go:122] "RemoveContainer" containerID="9e38753c060338b49e00edd4b78e366f8114eb7a2dea6f519ea7ad6949496044" Oct 30 02:15:06.643345 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:06.643322 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw_odh-ai-gateway-infra(0a25d75a-85d3-4f3f-a951-aa39f35481fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" podUID="0a25d75a-85d3-4f3f-a951-aa39f35481fb" Oct 30 02:15:06.646800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.646769 5998 generic.go:356] "Generic (PLEG): container finished" podID="7a692aea-70f7-4f61-adc6-bba42c9cc1bd" containerID="97d51b0a0f9445e428fb78d7b1a39c6a9c60f6fe4d593a4e60632a8f864fa5e1" exitCode=60 Oct 30 02:15:06.646907 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.646851 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" event={"ID":"7a692aea-70f7-4f61-adc6-bba42c9cc1bd","Type":"ContainerDied","Data":"97d51b0a0f9445e428fb78d7b1a39c6a9c60f6fe4d593a4e60632a8f864fa5e1"} Oct 30 02:15:06.646907 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.646893 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" event={"ID":"7a692aea-70f7-4f61-adc6-bba42c9cc1bd","Type":"ContainerStarted","Data":"312eeca8fa3280584b26b5e4d2bf53bf8eadde929380eb3c069cc77d852b40bf"} Oct 30 02:15:06.651959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.651929 5998 generic.go:356] "Generic (PLEG): container finished" podID="f70185f3-a978-4c11-ab66-903cfcba41b8" containerID="8aef59d64f6cca76cf005bc8fdeaf7a8aee19e6213b2d44be7799c9235feaf13" exitCode=60 Oct 30 02:15:06.652077 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.652034 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" event={"ID":"f70185f3-a978-4c11-ab66-903cfcba41b8","Type":"ContainerDied","Data":"8aef59d64f6cca76cf005bc8fdeaf7a8aee19e6213b2d44be7799c9235feaf13"} Oct 30 02:15:06.652192 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.652079 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" event={"ID":"f70185f3-a978-4c11-ab66-903cfcba41b8","Type":"ContainerStarted","Data":"745fa541097ce9c6aee6ca7b53a3184ef7043bd6a36f7afa6e5ee0565c2f4404"} Oct 30 02:15:06.661715 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.661686 5998 generic.go:356] "Generic (PLEG): container finished" podID="3212debe-7561-4fa1-bdd9-bd34e88389ea" containerID="0213e37e45852378455d942e474bd072504d6b685f87e004cc473529558fec7c" exitCode=60 Oct 30 02:15:06.662072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.661948 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" event={"ID":"3212debe-7561-4fa1-bdd9-bd34e88389ea","Type":"ContainerDied","Data":"0213e37e45852378455d942e474bd072504d6b685f87e004cc473529558fec7c"} Oct 30 02:15:06.669723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.669681 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" event={"ID":"3212debe-7561-4fa1-bdd9-bd34e88389ea","Type":"ContainerStarted","Data":"a5fad93a67bcc835cfbfaf58d7b1c6b2306aa700a57737df1eb612b4855491cd"} Oct 30 02:15:06.674927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.674882 5998 generic.go:356] "Generic (PLEG): container finished" podID="88d6d3ad-5fa9-4aca-9803-a981213c42af" containerID="a2eff0a6be011ac428c64992b7028eda977fe11eba491779da7f96823fb3b2cc" exitCode=60 Oct 30 02:15:06.675164 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.675018 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" event={"ID":"88d6d3ad-5fa9-4aca-9803-a981213c42af","Type":"ContainerDied","Data":"a2eff0a6be011ac428c64992b7028eda977fe11eba491779da7f96823fb3b2cc"} Oct 30 02:15:06.675164 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.675078 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" event={"ID":"88d6d3ad-5fa9-4aca-9803-a981213c42af","Type":"ContainerStarted","Data":"c1f4a20b044e34a91e770eae0aa1f7b71fb54c199b2c3ea6f6d7174852507b4f"} Oct 30 02:15:06.678323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.678225 5998 generic.go:356] "Generic (PLEG): container finished" podID="54e5a7ed-87b9-4be7-87fe-39944b774797" containerID="3d4a7bed67059fb9d2369d99edb7e8c618a55b9e17f21874f254c830e937f263" exitCode=60 Oct 30 02:15:06.681809 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.681449 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" event={"ID":"54e5a7ed-87b9-4be7-87fe-39944b774797","Type":"ContainerDied","Data":"3d4a7bed67059fb9d2369d99edb7e8c618a55b9e17f21874f254c830e937f263"} Oct 30 02:15:06.681994 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.681854 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" event={"ID":"54e5a7ed-87b9-4be7-87fe-39944b774797","Type":"ContainerStarted","Data":"76477a5993889172e4e1227acfe38c98bffa98e1f7fbb7ae8e53f9fb8ead48e6"} Oct 30 02:15:06.685068 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.685037 5998 generic.go:356] "Generic (PLEG): container finished" podID="bf51a9f7-5514-42fd-9122-75d1495cde92" containerID="535496fe2a36e44071e00874cbe975826ec6b6948a411ee524982a601da10d45" exitCode=60 Oct 30 02:15:06.685449 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.685428 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" event={"ID":"bf51a9f7-5514-42fd-9122-75d1495cde92","Type":"ContainerDied","Data":"535496fe2a36e44071e00874cbe975826ec6b6948a411ee524982a601da10d45"} Oct 30 02:15:06.685647 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.685631 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" event={"ID":"bf51a9f7-5514-42fd-9122-75d1495cde92","Type":"ContainerStarted","Data":"d96a1eb3afa8d37942552a6657fc807e568118b6eb4c7b583eb1331c76aeacd4"} Oct 30 02:15:06.690804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.690766 5998 generic.go:356] "Generic (PLEG): container finished" podID="30276c96-e1d6-4759-9368-51b819812537" containerID="56e1fad7611bfd4b3fad3a47a99feddef7d7811b92266ff989040c2ac6173e2d" exitCode=60 Oct 30 02:15:06.690936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.690897 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" event={"ID":"30276c96-e1d6-4759-9368-51b819812537","Type":"ContainerDied","Data":"56e1fad7611bfd4b3fad3a47a99feddef7d7811b92266ff989040c2ac6173e2d"} Oct 30 02:15:06.691016 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.690951 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" event={"ID":"30276c96-e1d6-4759-9368-51b819812537","Type":"ContainerStarted","Data":"03cb3c23ab3ca8318c0d3da6174d81e4e94919b2f930cdb2575f76609a42c9c4"} Oct 30 02:15:06.693968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.693936 5998 generic.go:356] "Generic (PLEG): container finished" podID="ab563a5a-c1f8-4e32-aa5d-91effabc8a18" containerID="6d7e2d396f865d312c14b29e06ceb9b7eb751c7cb4bb05b0ba0eac9f5bff46c6" exitCode=60 Oct 30 02:15:06.694072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.694024 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" event={"ID":"ab563a5a-c1f8-4e32-aa5d-91effabc8a18","Type":"ContainerDied","Data":"6d7e2d396f865d312c14b29e06ceb9b7eb751c7cb4bb05b0ba0eac9f5bff46c6"} Oct 30 02:15:06.694072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.694065 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" event={"ID":"ab563a5a-c1f8-4e32-aa5d-91effabc8a18","Type":"ContainerStarted","Data":"ea459951f930b24e11e61647e82ca5ae0975a4410418764ead4cc7d994554962"} Oct 30 02:15:06.699147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.699117 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7" event={"ID":"89f01e25-ada7-489b-966d-823f8bc623a1","Type":"ContainerStarted","Data":"1ad77659559040c1ce69d7006db85a594b73275b33541a63b43280390358dd06"} Oct 30 02:15:06.796597 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.796506 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" podStartSLOduration=6.7964712 podStartE2EDuration="6.7964712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:06.75724337 +0000 UTC m=+2736.336034452" watchObservedRunningTime="2026-10-30 02:15:06.7964712 +0000 UTC m=+2736.375262284" Oct 30 02:15:06.797262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.796926 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" podStartSLOduration=6.796919172 podStartE2EDuration="6.796919172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:06.795055689 +0000 UTC m=+2736.373846771" watchObservedRunningTime="2026-10-30 02:15:06.796919172 +0000 UTC m=+2736.375710253" Oct 30 02:15:06.922340 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:06.922279 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" podStartSLOduration=6.922260821 podStartE2EDuration="6.922260821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:06.915629503 +0000 UTC m=+2736.494420584" watchObservedRunningTime="2026-10-30 02:15:06.922260821 +0000 UTC m=+2736.501051905" Oct 30 02:15:07.047713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.047632 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" podStartSLOduration=7.047611464 podStartE2EDuration="7.047611464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:07.04029345 +0000 UTC m=+2736.619084534" watchObservedRunningTime="2026-10-30 02:15:07.047611464 +0000 UTC m=+2736.626402547" Oct 30 02:15:07.079933 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.079848 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" podStartSLOduration=7.079831451 podStartE2EDuration="7.079831451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:07.076733048 +0000 UTC m=+2736.655524130" watchObservedRunningTime="2026-10-30 02:15:07.079831451 +0000 UTC m=+2736.658622532" Oct 30 02:15:07.115906 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.115853 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" podStartSLOduration=7.11583738 podStartE2EDuration="7.11583738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:07.113115686 +0000 UTC m=+2736.691906770" watchObservedRunningTime="2026-10-30 02:15:07.11583738 +0000 UTC m=+2736.694628461" Oct 30 02:15:07.150770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.150710 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" podStartSLOduration=7.150696088 podStartE2EDuration="7.150696088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:07.150251845 +0000 UTC m=+2736.729042926" watchObservedRunningTime="2026-10-30 02:15:07.150696088 +0000 UTC m=+2736.729487168" Oct 30 02:15:07.189884 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.189826 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" podStartSLOduration=7.189811602 podStartE2EDuration="7.189811602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:07.186140722 +0000 UTC m=+2736.764931800" watchObservedRunningTime="2026-10-30 02:15:07.189811602 +0000 UTC m=+2736.768602682" Oct 30 02:15:07.242523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.242483 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:07.705164 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.705027 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw"] Oct 30 02:15:07.712273 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.712240 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.723195 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.721935 5998 generic.go:356] "Generic (PLEG): container finished" podID="ab563a5a-c1f8-4e32-aa5d-91effabc8a18" containerID="ea459951f930b24e11e61647e82ca5ae0975a4410418764ead4cc7d994554962" exitCode=60 Oct 30 02:15:07.723195 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.722034 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" event={"ID":"ab563a5a-c1f8-4e32-aa5d-91effabc8a18","Type":"ContainerDied","Data":"ea459951f930b24e11e61647e82ca5ae0975a4410418764ead4cc7d994554962"} Oct 30 02:15:07.723195 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.722113 5998 scope.go:122] "RemoveContainer" containerID="6d7e2d396f865d312c14b29e06ceb9b7eb751c7cb4bb05b0ba0eac9f5bff46c6" Oct 30 02:15:07.723195 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.722457 5998 scope.go:122] "RemoveContainer" containerID="ea459951f930b24e11e61647e82ca5ae0975a4410418764ead4cc7d994554962" Oct 30 02:15:07.723195 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:07.722875 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr_odh-ai-gateway-infra(ab563a5a-c1f8-4e32-aa5d-91effabc8a18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" podUID="ab563a5a-c1f8-4e32-aa5d-91effabc8a18" Oct 30 02:15:07.727394 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.727365 5998 generic.go:356] "Generic (PLEG): container finished" podID="89f01e25-ada7-489b-966d-823f8bc623a1" containerID="e72fac36fe6f8e02dec78f560cc2e0227b538576bd0060b5266820ea16895399" exitCode=0 Oct 30 02:15:07.727582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.727559 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7" event={"ID":"89f01e25-ada7-489b-966d-823f8bc623a1","Type":"ContainerDied","Data":"e72fac36fe6f8e02dec78f560cc2e0227b538576bd0060b5266820ea16895399"} Oct 30 02:15:07.730355 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.730326 5998 generic.go:356] "Generic (PLEG): container finished" podID="a93eb477-297b-4060-a79a-8ba36de67e83" containerID="5e746fcfb42b8f50700442a24860f6ca573d5e0ac79beea3d9a47505707c7a45" exitCode=60 Oct 30 02:15:07.730496 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.730431 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" event={"ID":"a93eb477-297b-4060-a79a-8ba36de67e83","Type":"ContainerDied","Data":"5e746fcfb42b8f50700442a24860f6ca573d5e0ac79beea3d9a47505707c7a45"} Oct 30 02:15:07.730954 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.730918 5998 scope.go:122] "RemoveContainer" containerID="5e746fcfb42b8f50700442a24860f6ca573d5e0ac79beea3d9a47505707c7a45" Oct 30 02:15:07.731536 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:07.731367 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2_odh-ai-gateway-infra(a93eb477-297b-4060-a79a-8ba36de67e83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" podUID="a93eb477-297b-4060-a79a-8ba36de67e83" Oct 30 02:15:07.733595 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.733567 5998 generic.go:356] "Generic (PLEG): container finished" podID="ea963033-f2cb-4e1c-a9ef-7d25c76581eb" containerID="4bd0d577b2c02c1076354dca6905804599b930fef79bcfce3235f7aaea5b4187" exitCode=60 Oct 30 02:15:07.733735 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.733676 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" event={"ID":"ea963033-f2cb-4e1c-a9ef-7d25c76581eb","Type":"ContainerDied","Data":"4bd0d577b2c02c1076354dca6905804599b930fef79bcfce3235f7aaea5b4187"} Oct 30 02:15:07.734289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.734240 5998 scope.go:122] "RemoveContainer" containerID="4bd0d577b2c02c1076354dca6905804599b930fef79bcfce3235f7aaea5b4187" Oct 30 02:15:07.734689 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:07.734663 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888775-xz57t_odh-ai-gateway-infra(ea963033-f2cb-4e1c-a9ef-7d25c76581eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" podUID="ea963033-f2cb-4e1c-a9ef-7d25c76581eb" Oct 30 02:15:07.742426 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.742388 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:07.744268 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.744241 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw"] Oct 30 02:15:07.747785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.747759 5998 scope.go:122] "RemoveContainer" containerID="9e38753c060338b49e00edd4b78e366f8114eb7a2dea6f519ea7ad6949496044" Oct 30 02:15:07.748211 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:07.748178 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw_odh-ai-gateway-infra(0a25d75a-85d3-4f3f-a951-aa39f35481fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" podUID="0a25d75a-85d3-4f3f-a951-aa39f35481fb" Oct 30 02:15:07.749954 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.749923 5998 generic.go:356] "Generic (PLEG): container finished" podID="7a692aea-70f7-4f61-adc6-bba42c9cc1bd" containerID="312eeca8fa3280584b26b5e4d2bf53bf8eadde929380eb3c069cc77d852b40bf" exitCode=60 Oct 30 02:15:07.750075 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.750011 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" event={"ID":"7a692aea-70f7-4f61-adc6-bba42c9cc1bd","Type":"ContainerDied","Data":"312eeca8fa3280584b26b5e4d2bf53bf8eadde929380eb3c069cc77d852b40bf"} Oct 30 02:15:07.750594 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.750548 5998 scope.go:122] "RemoveContainer" containerID="312eeca8fa3280584b26b5e4d2bf53bf8eadde929380eb3c069cc77d852b40bf" Oct 30 02:15:07.750935 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:07.750911 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc_odh-ai-gateway-infra(7a692aea-70f7-4f61-adc6-bba42c9cc1bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" podUID="7a692aea-70f7-4f61-adc6-bba42c9cc1bd" Oct 30 02:15:07.753078 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.753057 5998 generic.go:356] "Generic (PLEG): container finished" podID="f70185f3-a978-4c11-ab66-903cfcba41b8" containerID="745fa541097ce9c6aee6ca7b53a3184ef7043bd6a36f7afa6e5ee0565c2f4404" exitCode=60 Oct 30 02:15:07.753625 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.753607 5998 scope.go:122] "RemoveContainer" containerID="745fa541097ce9c6aee6ca7b53a3184ef7043bd6a36f7afa6e5ee0565c2f4404" Oct 30 02:15:07.754037 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:07.754015 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s_odh-ai-gateway-infra(f70185f3-a978-4c11-ab66-903cfcba41b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" podUID="f70185f3-a978-4c11-ab66-903cfcba41b8" Oct 30 02:15:07.754233 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.754213 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" event={"ID":"f70185f3-a978-4c11-ab66-903cfcba41b8","Type":"ContainerDied","Data":"745fa541097ce9c6aee6ca7b53a3184ef7043bd6a36f7afa6e5ee0565c2f4404"} Oct 30 02:15:07.756004 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.755981 5998 generic.go:356] "Generic (PLEG): container finished" podID="3212debe-7561-4fa1-bdd9-bd34e88389ea" containerID="a5fad93a67bcc835cfbfaf58d7b1c6b2306aa700a57737df1eb612b4855491cd" exitCode=60 Oct 30 02:15:07.756141 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.756121 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" event={"ID":"3212debe-7561-4fa1-bdd9-bd34e88389ea","Type":"ContainerDied","Data":"a5fad93a67bcc835cfbfaf58d7b1c6b2306aa700a57737df1eb612b4855491cd"} Oct 30 02:15:07.757606 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.757583 5998 scope.go:122] "RemoveContainer" containerID="a5fad93a67bcc835cfbfaf58d7b1c6b2306aa700a57737df1eb612b4855491cd" Oct 30 02:15:07.757920 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:07.757899 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s_odh-ai-gateway-infra(3212debe-7561-4fa1-bdd9-bd34e88389ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" podUID="3212debe-7561-4fa1-bdd9-bd34e88389ea" Oct 30 02:15:07.764183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.764152 5998 generic.go:356] "Generic (PLEG): container finished" podID="88d6d3ad-5fa9-4aca-9803-a981213c42af" containerID="c1f4a20b044e34a91e770eae0aa1f7b71fb54c199b2c3ea6f6d7174852507b4f" exitCode=60 Oct 30 02:15:07.764349 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.764251 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" event={"ID":"88d6d3ad-5fa9-4aca-9803-a981213c42af","Type":"ContainerDied","Data":"c1f4a20b044e34a91e770eae0aa1f7b71fb54c199b2c3ea6f6d7174852507b4f"} Oct 30 02:15:07.764857 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.764588 5998 scope.go:122] "RemoveContainer" containerID="c1f4a20b044e34a91e770eae0aa1f7b71fb54c199b2c3ea6f6d7174852507b4f" Oct 30 02:15:07.764993 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:07.764969 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb_odh-ai-gateway-infra(88d6d3ad-5fa9-4aca-9803-a981213c42af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" podUID="88d6d3ad-5fa9-4aca-9803-a981213c42af" Oct 30 02:15:07.774342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.774288 5998 scope.go:122] "RemoveContainer" containerID="76201069ee9a86ecb54c6711fa6c3f888ab77b42ecf9cdeec5fcbd63473a4f27" Oct 30 02:15:07.774601 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.774578 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-0ce950-openshift-default-dockercfg-hzmm5\"" Oct 30 02:15:07.777429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.777402 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" event={"ID":"54e5a7ed-87b9-4be7-87fe-39944b774797","Type":"ContainerDied","Data":"76477a5993889172e4e1227acfe38c98bffa98e1f7fbb7ae8e53f9fb8ead48e6"} Oct 30 02:15:07.777597 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.777573 5998 scope.go:122] "RemoveContainer" containerID="76477a5993889172e4e1227acfe38c98bffa98e1f7fbb7ae8e53f9fb8ead48e6" Oct 30 02:15:07.778046 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:07.778020 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms_odh-ai-gateway-infra(54e5a7ed-87b9-4be7-87fe-39944b774797)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" podUID="54e5a7ed-87b9-4be7-87fe-39944b774797" Oct 30 02:15:07.778144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.778112 5998 generic.go:356] "Generic (PLEG): container finished" podID="54e5a7ed-87b9-4be7-87fe-39944b774797" containerID="76477a5993889172e4e1227acfe38c98bffa98e1f7fbb7ae8e53f9fb8ead48e6" exitCode=60 Oct 30 02:15:07.780892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.780869 5998 generic.go:356] "Generic (PLEG): container finished" podID="bf51a9f7-5514-42fd-9122-75d1495cde92" containerID="d96a1eb3afa8d37942552a6657fc807e568118b6eb4c7b583eb1331c76aeacd4" exitCode=60 Oct 30 02:15:07.781035 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.780993 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" event={"ID":"bf51a9f7-5514-42fd-9122-75d1495cde92","Type":"ContainerDied","Data":"d96a1eb3afa8d37942552a6657fc807e568118b6eb4c7b583eb1331c76aeacd4"} Oct 30 02:15:07.781498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.781474 5998 scope.go:122] "RemoveContainer" containerID="d96a1eb3afa8d37942552a6657fc807e568118b6eb4c7b583eb1331c76aeacd4" Oct 30 02:15:07.781863 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:07.781826 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb_odh-ai-gateway-infra(bf51a9f7-5514-42fd-9122-75d1495cde92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" podUID="bf51a9f7-5514-42fd-9122-75d1495cde92" Oct 30 02:15:07.784661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.784637 5998 generic.go:356] "Generic (PLEG): container finished" podID="30276c96-e1d6-4759-9368-51b819812537" containerID="03cb3c23ab3ca8318c0d3da6174d81e4e94919b2f930cdb2575f76609a42c9c4" exitCode=60 Oct 30 02:15:07.785874 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.785857 5998 scope.go:122] "RemoveContainer" containerID="03cb3c23ab3ca8318c0d3da6174d81e4e94919b2f930cdb2575f76609a42c9c4" Oct 30 02:15:07.786220 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:07.786188 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf_odh-ai-gateway-infra(30276c96-e1d6-4759-9368-51b819812537)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" podUID="30276c96-e1d6-4759-9368-51b819812537" Oct 30 02:15:07.786308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.786246 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" event={"ID":"30276c96-e1d6-4759-9368-51b819812537","Type":"ContainerDied","Data":"03cb3c23ab3ca8318c0d3da6174d81e4e94919b2f930cdb2575f76609a42c9c4"} Oct 30 02:15:07.795396 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.795372 5998 scope.go:122] "RemoveContainer" containerID="aea588db8423e78647f018db2b24bfec6520f320db5fd868db8ffdc13463adc2" Oct 30 02:15:07.861548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.861505 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-token\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.861735 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.861651 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istiod-ca-cert\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.861735 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.861719 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-data\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.861886 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.861815 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-credential-socket\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.862255 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.862161 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:07.863127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.863059 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-envoy\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.863906 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.863753 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-podinfo\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.863906 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.863792 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-workload-socket\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.864145 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.863932 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-workload-certs\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.864145 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.864023 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kljp\" (UniqueName: \"kubernetes.io/projected/d71cb5c7-54c2-42ed-9d33-de325c6c851a-kube-api-access-5kljp\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.880350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.880322 5998 scope.go:122] "RemoveContainer" containerID="97d51b0a0f9445e428fb78d7b1a39c6a9c60f6fe4d593a4e60632a8f864fa5e1" Oct 30 02:15:07.903958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.903931 5998 scope.go:122] "RemoveContainer" containerID="8aef59d64f6cca76cf005bc8fdeaf7a8aee19e6213b2d44be7799c9235feaf13" Oct 30 02:15:07.919600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.919571 5998 scope.go:122] "RemoveContainer" containerID="0213e37e45852378455d942e474bd072504d6b685f87e004cc473529558fec7c" Oct 30 02:15:07.938253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.938227 5998 scope.go:122] "RemoveContainer" containerID="a2eff0a6be011ac428c64992b7028eda977fe11eba491779da7f96823fb3b2cc" Oct 30 02:15:07.953931 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.953805 5998 scope.go:122] "RemoveContainer" containerID="3d4a7bed67059fb9d2369d99edb7e8c618a55b9e17f21874f254c830e937f263" Oct 30 02:15:07.965743 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.965667 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-workload-certs\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.965743 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.965719 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kljp\" (UniqueName: \"kubernetes.io/projected/d71cb5c7-54c2-42ed-9d33-de325c6c851a-kube-api-access-5kljp\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.965958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.965746 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-token\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.965958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.965788 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istiod-ca-cert\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.965958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.965834 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-data\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.965958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.965898 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-credential-socket\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.966217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.965977 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-envoy\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.966217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.966168 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-workload-certs\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.966217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.966174 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-podinfo\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.966381 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.966247 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-workload-socket\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.966454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.966385 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-data\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.966730 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.966703 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-workload-socket\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.966730 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.966722 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-credential-socket\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.966866 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.966712 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istiod-ca-cert\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.969067 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.969037 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-envoy\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.969503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.969484 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-podinfo\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:07.977631 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.977292 5998 scope.go:122] "RemoveContainer" containerID="535496fe2a36e44071e00874cbe975826ec6b6948a411ee524982a601da10d45" Oct 30 02:15:07.978263 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:07.978224 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:08.100172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.084353 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-token\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:08.169116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.150838 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kljp\" (UniqueName: \"kubernetes.io/projected/d71cb5c7-54c2-42ed-9d33-de325c6c851a-kube-api-access-5kljp\") pod \"e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:08.276323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.275581 5998 scope.go:122] "RemoveContainer" containerID="56e1fad7611bfd4b3fad3a47a99feddef7d7811b92266ff989040c2ac6173e2d" Oct 30 02:15:08.287231 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.287207 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:15:08.326253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.326215 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-data\") pod \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " Oct 30 02:15:08.326452 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.326287 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-workload-socket\") pod \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " Oct 30 02:15:08.326452 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.326356 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-credential-socket\") pod \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " Oct 30 02:15:08.326452 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.326423 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-workload-certs\") pod \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " Oct 30 02:15:08.326452 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.326434 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-data" pod "1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" (UID: "1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:08.326683 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.326484 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istiod-ca-cert\") pod \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " Oct 30 02:15:08.326683 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.326576 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-podinfo\") pod \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " Oct 30 02:15:08.326683 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.326604 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-workload-socket" pod "1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" (UID: "1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:08.326849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.326698 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-token\" (UniqueName: \"kubernetes.io/projected/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-token\") pod \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " Oct 30 02:15:08.326849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.326733 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-kube-api-access-6sbhc\" (UniqueName: \"kubernetes.io/projected/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-kube-api-access-6sbhc\") pod \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " Oct 30 02:15:08.326849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.326758 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-credential-socket" pod "1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" (UID: "1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:08.326849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.326789 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-workload-certs" pod "1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" (UID: "1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:08.326849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.326834 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-envoy\") pod \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\" (UID: \"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e\") " Oct 30 02:15:08.327363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.327329 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istiod-ca-cert" pod "1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" (UID: "1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:08.327639 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.327621 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:08.327745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.327643 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:08.327745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.327658 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:08.327745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.327670 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:08.329297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.329277 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-podinfo" pod "1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" (UID: "1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:15:08.329433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.329403 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-envoy" pod "1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" (UID: "1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:08.329527 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.329506 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-token" pod "1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" (UID: "1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:08.329591 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.329538 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-kube-api-access-6sbhc" pod "1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" (UID: "1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e"). InnerVolumeSpecName "kube-api-access-6sbhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:08.347760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.347724 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:08.367175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.367139 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:08.431377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.430075 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:08.431377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.430124 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:08.431377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.430141 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:08.431377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.430159 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:08.431377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.430173 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sbhc\" (UniqueName: \"kubernetes.io/projected/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e-kube-api-access-6sbhc\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:08.482115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.478903 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:08.714128 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:15:08.714072 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71cb5c7_54c2_42ed_9d33_de325c6c851a.slice/crio-fea45f3f4fef9f785ddcb3f343b0c1fba793367923cc2521671eaedf1d2c9d44 WatchSource:0}: Error finding container fea45f3f4fef9f785ddcb3f343b0c1fba793367923cc2521671eaedf1d2c9d44: Status 404 returned error can't find the container with id fea45f3f4fef9f785ddcb3f343b0c1fba793367923cc2521671eaedf1d2c9d44 Oct 30 02:15:08.715829 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.715799 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw"] Oct 30 02:15:08.717682 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.717638 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:15:08.717775 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.717736 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:15:08.717820 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.717778 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:15:08.802391 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.802293 5998 generic.go:356] "Generic (PLEG): container finished" podID="287b0a33-1dc7-4284-8c93-7af661cf9d37" containerID="9094fc14d31491d82ba094f4bcfc217786beadf16f6baaa3f80552a1603d59d1" exitCode=60 Oct 30 02:15:08.802575 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.802442 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" event={"ID":"287b0a33-1dc7-4284-8c93-7af661cf9d37","Type":"ContainerDied","Data":"9094fc14d31491d82ba094f4bcfc217786beadf16f6baaa3f80552a1603d59d1"} Oct 30 02:15:08.802575 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.802503 5998 scope.go:122] "RemoveContainer" containerID="307ee30e30037fc5821854104b6ffcdbf81ac61734089387493384debcd82eb6" Oct 30 02:15:08.802882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.802856 5998 scope.go:122] "RemoveContainer" containerID="9094fc14d31491d82ba094f4bcfc217786beadf16f6baaa3f80552a1603d59d1" Oct 30 02:15:08.803688 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:08.803657 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7_odh-ai-gateway-infra(287b0a33-1dc7-4284-8c93-7af661cf9d37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" podUID="287b0a33-1dc7-4284-8c93-7af661cf9d37" Oct 30 02:15:08.805591 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.805568 5998 generic.go:356] "Generic (PLEG): container finished" podID="3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4" containerID="21c065a8ec10db0b25e3e35a529a7210cefe9874cc1c7b6ea5e8f0b804c47e58" exitCode=60 Oct 30 02:15:08.805781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.805660 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" event={"ID":"3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4","Type":"ContainerDied","Data":"21c065a8ec10db0b25e3e35a529a7210cefe9874cc1c7b6ea5e8f0b804c47e58"} Oct 30 02:15:08.806118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.805970 5998 scope.go:122] "RemoveContainer" containerID="21c065a8ec10db0b25e3e35a529a7210cefe9874cc1c7b6ea5e8f0b804c47e58" Oct 30 02:15:08.806413 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:08.806386 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr_odh-ai-gateway-infra(3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" podUID="3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4" Oct 30 02:15:08.809879 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.809808 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" event={"ID":"d71cb5c7-54c2-42ed-9d33-de325c6c851a","Type":"ContainerStarted","Data":"fea45f3f4fef9f785ddcb3f343b0c1fba793367923cc2521671eaedf1d2c9d44"} Oct 30 02:15:08.821165 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.820706 5998 scope.go:122] "RemoveContainer" containerID="5e746fcfb42b8f50700442a24860f6ca573d5e0ac79beea3d9a47505707c7a45" Oct 30 02:15:08.822190 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:08.822163 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2_odh-ai-gateway-infra(a93eb477-297b-4060-a79a-8ba36de67e83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" podUID="a93eb477-297b-4060-a79a-8ba36de67e83" Oct 30 02:15:08.824275 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.824229 5998 scope.go:122] "RemoveContainer" containerID="4bd0d577b2c02c1076354dca6905804599b930fef79bcfce3235f7aaea5b4187" Oct 30 02:15:08.824585 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:08.824558 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888775-xz57t_odh-ai-gateway-infra(ea963033-f2cb-4e1c-a9ef-7d25c76581eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" podUID="ea963033-f2cb-4e1c-a9ef-7d25c76581eb" Oct 30 02:15:08.826677 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.826656 5998 scope.go:122] "RemoveContainer" containerID="fa8291a4d07f31b776be6da55ecdc3cb0f37bb0aa1e2c23f180b0449fd31c9c7" Oct 30 02:15:08.827316 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.827300 5998 scope.go:122] "RemoveContainer" containerID="312eeca8fa3280584b26b5e4d2bf53bf8eadde929380eb3c069cc77d852b40bf" Oct 30 02:15:08.827597 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:08.827575 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc_odh-ai-gateway-infra(7a692aea-70f7-4f61-adc6-bba42c9cc1bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" podUID="7a692aea-70f7-4f61-adc6-bba42c9cc1bd" Oct 30 02:15:08.829847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.829828 5998 scope.go:122] "RemoveContainer" containerID="745fa541097ce9c6aee6ca7b53a3184ef7043bd6a36f7afa6e5ee0565c2f4404" Oct 30 02:15:08.830485 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:08.830137 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s_odh-ai-gateway-infra(f70185f3-a978-4c11-ab66-903cfcba41b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" podUID="f70185f3-a978-4c11-ab66-903cfcba41b8" Oct 30 02:15:08.833109 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.833062 5998 scope.go:122] "RemoveContainer" containerID="a5fad93a67bcc835cfbfaf58d7b1c6b2306aa700a57737df1eb612b4855491cd" Oct 30 02:15:08.833490 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:08.833467 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s_odh-ai-gateway-infra(3212debe-7561-4fa1-bdd9-bd34e88389ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" podUID="3212debe-7561-4fa1-bdd9-bd34e88389ea" Oct 30 02:15:08.835016 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.834993 5998 scope.go:122] "RemoveContainer" containerID="c1f4a20b044e34a91e770eae0aa1f7b71fb54c199b2c3ea6f6d7174852507b4f" Oct 30 02:15:08.835297 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:08.835275 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb_odh-ai-gateway-infra(88d6d3ad-5fa9-4aca-9803-a981213c42af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" podUID="88d6d3ad-5fa9-4aca-9803-a981213c42af" Oct 30 02:15:08.837722 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.837697 5998 scope.go:122] "RemoveContainer" containerID="76477a5993889172e4e1227acfe38c98bffa98e1f7fbb7ae8e53f9fb8ead48e6" Oct 30 02:15:08.838029 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:08.838005 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms_odh-ai-gateway-infra(54e5a7ed-87b9-4be7-87fe-39944b774797)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" podUID="54e5a7ed-87b9-4be7-87fe-39944b774797" Oct 30 02:15:08.849118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.848687 5998 scope.go:122] "RemoveContainer" containerID="03cb3c23ab3ca8318c0d3da6174d81e4e94919b2f930cdb2575f76609a42c9c4" Oct 30 02:15:08.849118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.848887 5998 generic.go:356] "Generic (PLEG): container finished" podID="1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" containerID="cb0b55c49f2ada5676d0d7b8de4dcc9cf8ff84c1ac3da34f992344ce5e0cf39b" exitCode=0 Oct 30 02:15:08.849118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.848950 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" Oct 30 02:15:08.849118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.848974 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" event={"ID":"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e","Type":"ContainerDied","Data":"cb0b55c49f2ada5676d0d7b8de4dcc9cf8ff84c1ac3da34f992344ce5e0cf39b"} Oct 30 02:15:08.849118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.849033 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph" event={"ID":"1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e","Type":"ContainerDied","Data":"564c523a964f8d4d8f373e54be2b60badbf3efe46b096c94b468cc325e70a3e3"} Oct 30 02:15:08.849118 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:08.849045 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf_odh-ai-gateway-infra(30276c96-e1d6-4759-9368-51b819812537)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" podUID="30276c96-e1d6-4759-9368-51b819812537" Oct 30 02:15:08.849118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.849063 5998 scope.go:122] "RemoveContainer" containerID="cb0b55c49f2ada5676d0d7b8de4dcc9cf8ff84c1ac3da34f992344ce5e0cf39b" Oct 30 02:15:08.854800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.853901 5998 scope.go:122] "RemoveContainer" containerID="d96a1eb3afa8d37942552a6657fc807e568118b6eb4c7b583eb1331c76aeacd4" Oct 30 02:15:08.854800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.854358 5998 scope.go:122] "RemoveContainer" containerID="ea459951f930b24e11e61647e82ca5ae0975a4410418764ead4cc7d994554962" Oct 30 02:15:08.854800 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:08.854626 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr_odh-ai-gateway-infra(ab563a5a-c1f8-4e32-aa5d-91effabc8a18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" podUID="ab563a5a-c1f8-4e32-aa5d-91effabc8a18" Oct 30 02:15:08.854800 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:08.854671 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb_odh-ai-gateway-infra(bf51a9f7-5514-42fd-9122-75d1495cde92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" podUID="bf51a9f7-5514-42fd-9122-75d1495cde92" Oct 30 02:15:08.878023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.877553 5998 scope.go:122] "RemoveContainer" containerID="cb0b55c49f2ada5676d0d7b8de4dcc9cf8ff84c1ac3da34f992344ce5e0cf39b" Oct 30 02:15:08.878353 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:08.878252 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0b55c49f2ada5676d0d7b8de4dcc9cf8ff84c1ac3da34f992344ce5e0cf39b\": container with ID starting with cb0b55c49f2ada5676d0d7b8de4dcc9cf8ff84c1ac3da34f992344ce5e0cf39b not found: ID does not exist" containerID="cb0b55c49f2ada5676d0d7b8de4dcc9cf8ff84c1ac3da34f992344ce5e0cf39b" Oct 30 02:15:08.878353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:08.878291 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0b55c49f2ada5676d0d7b8de4dcc9cf8ff84c1ac3da34f992344ce5e0cf39b"} err="failed to get container status \"cb0b55c49f2ada5676d0d7b8de4dcc9cf8ff84c1ac3da34f992344ce5e0cf39b\": rpc error: code = NotFound desc = could not find container \"cb0b55c49f2ada5676d0d7b8de4dcc9cf8ff84c1ac3da34f992344ce5e0cf39b\": container with ID starting with cb0b55c49f2ada5676d0d7b8de4dcc9cf8ff84c1ac3da34f992344ce5e0cf39b not found: ID does not exist" Oct 30 02:15:09.233759 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.233663 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7" Oct 30 02:15:09.359408 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.359370 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89f01e25-ada7-489b-966d-823f8bc623a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f01e25-ada7-489b-966d-823f8bc623a1-maas-api-xks-ca\") pod \"89f01e25-ada7-489b-966d-823f8bc623a1\" (UID: \"89f01e25-ada7-489b-966d-823f8bc623a1\") " Oct 30 02:15:09.359771 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.359635 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f01e25-ada7-489b-966d-823f8bc623a1-kube-api-access-k5vsw\" (UniqueName: \"kubernetes.io/projected/89f01e25-ada7-489b-966d-823f8bc623a1-kube-api-access-k5vsw\") pod \"89f01e25-ada7-489b-966d-823f8bc623a1\" (UID: \"89f01e25-ada7-489b-966d-823f8bc623a1\") " Oct 30 02:15:09.359872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.359846 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89f01e25-ada7-489b-966d-823f8bc623a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f01e25-ada7-489b-966d-823f8bc623a1-maas-api-openshift-ca\") pod \"89f01e25-ada7-489b-966d-823f8bc623a1\" (UID: \"89f01e25-ada7-489b-966d-823f8bc623a1\") " Oct 30 02:15:09.361102 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.361055 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f01e25-ada7-489b-966d-823f8bc623a1-maas-api-openshift-ca" pod "89f01e25-ada7-489b-966d-823f8bc623a1" (UID: "89f01e25-ada7-489b-966d-823f8bc623a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:09.362055 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.362017 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89f01e25-ada7-489b-966d-823f8bc623a1-maas-api-xks-ca" pod "89f01e25-ada7-489b-966d-823f8bc623a1" (UID: "89f01e25-ada7-489b-966d-823f8bc623a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:15:09.363524 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.363494 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f01e25-ada7-489b-966d-823f8bc623a1-kube-api-access-k5vsw" pod "89f01e25-ada7-489b-966d-823f8bc623a1" (UID: "89f01e25-ada7-489b-966d-823f8bc623a1"). InnerVolumeSpecName "kube-api-access-k5vsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:09.400460 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.400422 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph"] Oct 30 02:15:09.406595 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.406562 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-2hnph"] Oct 30 02:15:09.462808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.462734 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f01e25-ada7-489b-966d-823f8bc623a1-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:09.462808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.462772 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5vsw\" (UniqueName: \"kubernetes.io/projected/89f01e25-ada7-489b-966d-823f8bc623a1-kube-api-access-k5vsw\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:09.462808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.462807 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f01e25-ada7-489b-966d-823f8bc623a1-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:09.804847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.804759 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:09.882958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.882927 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7" Oct 30 02:15:09.882958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.882943 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7" event={"ID":"89f01e25-ada7-489b-966d-823f8bc623a1","Type":"ContainerDied","Data":"1ad77659559040c1ce69d7006db85a594b73275b33541a63b43280390358dd06"} Oct 30 02:15:09.883229 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.882992 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ad77659559040c1ce69d7006db85a594b73275b33541a63b43280390358dd06" Oct 30 02:15:09.885438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.885413 5998 scope.go:122] "RemoveContainer" containerID="9094fc14d31491d82ba094f4bcfc217786beadf16f6baaa3f80552a1603d59d1" Oct 30 02:15:09.885747 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:09.885720 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7_odh-ai-gateway-infra(287b0a33-1dc7-4284-8c93-7af661cf9d37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" podUID="287b0a33-1dc7-4284-8c93-7af661cf9d37" Oct 30 02:15:09.888098 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.888066 5998 scope.go:122] "RemoveContainer" containerID="21c065a8ec10db0b25e3e35a529a7210cefe9874cc1c7b6ea5e8f0b804c47e58" Oct 30 02:15:09.888433 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:09.888403 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr_odh-ai-gateway-infra(3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" podUID="3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4" Oct 30 02:15:09.889835 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:09.889812 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" event={"ID":"d71cb5c7-54c2-42ed-9d33-de325c6c851a","Type":"ContainerStarted","Data":"ec6c449f8666fee5516ac536d6fc4ee05fe5053a80fe1061479126cf82b278b1"} Oct 30 02:15:10.023016 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:10.022947 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" podStartSLOduration=3.022926433 podStartE2EDuration="3.022926433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:10.016621582 +0000 UTC m=+2739.595412657" watchObservedRunningTime="2026-10-30 02:15:10.022926433 +0000 UTC m=+2739.601717515" Oct 30 02:15:10.348862 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:10.348826 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:10.354962 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:10.354929 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:10.540506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:10.540472 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" path="/var/lib/kubelet/pods/1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e/volumes" Oct 30 02:15:10.827294 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:10.827259 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:10.899515 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:10.899480 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:10.900486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:10.900464 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:15:10.937003 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:10.936965 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:11.744270 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:11.744232 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:12.455246 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:12.455207 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:12.567953 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:12.567920 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:13.486473 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:13.486429 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:14.214113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:14.207807 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:14.314512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:14.314478 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:15.308252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:15.308212 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:16.839550 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:16.839512 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq"] Oct 30 02:15:16.866156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:16.866120 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:16.993795 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:16.993761 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:17.532431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:17.532397 5998 scope.go:122] "RemoveContainer" containerID="76477a5993889172e4e1227acfe38c98bffa98e1f7fbb7ae8e53f9fb8ead48e6" Oct 30 02:15:17.532659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:17.532526 5998 scope.go:122] "RemoveContainer" containerID="4bd0d577b2c02c1076354dca6905804599b930fef79bcfce3235f7aaea5b4187" Oct 30 02:15:17.991873 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:17.991832 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" event={"ID":"ea963033-f2cb-4e1c-a9ef-7d25c76581eb","Type":"ContainerStarted","Data":"a97fb8e717c145ea90c2760afcfee37f7e8a832b710a731fe0ffa7458d297381"} Oct 30 02:15:17.994173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:17.994148 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" event={"ID":"54e5a7ed-87b9-4be7-87fe-39944b774797","Type":"ContainerStarted","Data":"ee603f10c2aaefe0ac834ca58bda70fb1e414d7defc830a288f8b26591e3fede"} Oct 30 02:15:18.532619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:18.532589 5998 scope.go:122] "RemoveContainer" containerID="a5fad93a67bcc835cfbfaf58d7b1c6b2306aa700a57737df1eb612b4855491cd" Oct 30 02:15:18.533418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:18.533395 5998 scope.go:122] "RemoveContainer" containerID="9e38753c060338b49e00edd4b78e366f8114eb7a2dea6f519ea7ad6949496044" Oct 30 02:15:18.533554 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:18.533537 5998 scope.go:122] "RemoveContainer" containerID="03cb3c23ab3ca8318c0d3da6174d81e4e94919b2f930cdb2575f76609a42c9c4" Oct 30 02:15:18.533825 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:18.533810 5998 scope.go:122] "RemoveContainer" containerID="5e746fcfb42b8f50700442a24860f6ca573d5e0ac79beea3d9a47505707c7a45" Oct 30 02:15:18.534028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:18.534011 5998 scope.go:122] "RemoveContainer" containerID="312eeca8fa3280584b26b5e4d2bf53bf8eadde929380eb3c069cc77d852b40bf" Oct 30 02:15:18.534116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:18.534057 5998 scope.go:122] "RemoveContainer" containerID="c1f4a20b044e34a91e770eae0aa1f7b71fb54c199b2c3ea6f6d7174852507b4f" Oct 30 02:15:18.774567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:18.774536 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:19.007057 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:19.007020 5998 generic.go:356] "Generic (PLEG): container finished" podID="ea963033-f2cb-4e1c-a9ef-7d25c76581eb" containerID="a97fb8e717c145ea90c2760afcfee37f7e8a832b710a731fe0ffa7458d297381" exitCode=60 Oct 30 02:15:19.007613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:19.007107 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" event={"ID":"ea963033-f2cb-4e1c-a9ef-7d25c76581eb","Type":"ContainerDied","Data":"a97fb8e717c145ea90c2760afcfee37f7e8a832b710a731fe0ffa7458d297381"} Oct 30 02:15:19.007613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:19.007177 5998 scope.go:122] "RemoveContainer" containerID="4bd0d577b2c02c1076354dca6905804599b930fef79bcfce3235f7aaea5b4187" Oct 30 02:15:19.007720 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:19.007626 5998 scope.go:122] "RemoveContainer" containerID="a97fb8e717c145ea90c2760afcfee37f7e8a832b710a731fe0ffa7458d297381" Oct 30 02:15:19.008007 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:19.007983 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888775-xz57t_odh-ai-gateway-infra(ea963033-f2cb-4e1c-a9ef-7d25c76581eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" podUID="ea963033-f2cb-4e1c-a9ef-7d25c76581eb" Oct 30 02:15:19.009840 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:19.009818 5998 generic.go:356] "Generic (PLEG): container finished" podID="54e5a7ed-87b9-4be7-87fe-39944b774797" containerID="ee603f10c2aaefe0ac834ca58bda70fb1e414d7defc830a288f8b26591e3fede" exitCode=60 Oct 30 02:15:19.010043 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:19.009859 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" event={"ID":"54e5a7ed-87b9-4be7-87fe-39944b774797","Type":"ContainerDied","Data":"ee603f10c2aaefe0ac834ca58bda70fb1e414d7defc830a288f8b26591e3fede"} Oct 30 02:15:19.010290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:19.010271 5998 scope.go:122] "RemoveContainer" containerID="ee603f10c2aaefe0ac834ca58bda70fb1e414d7defc830a288f8b26591e3fede" Oct 30 02:15:19.010618 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:19.010597 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms_odh-ai-gateway-infra(54e5a7ed-87b9-4be7-87fe-39944b774797)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" podUID="54e5a7ed-87b9-4be7-87fe-39944b774797" Oct 30 02:15:19.023889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:19.023851 5998 scope.go:122] "RemoveContainer" containerID="76477a5993889172e4e1227acfe38c98bffa98e1f7fbb7ae8e53f9fb8ead48e6" Oct 30 02:15:19.041393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:19.041364 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t"] Oct 30 02:15:19.089986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:19.089944 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms"] Oct 30 02:15:19.532678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:19.532498 5998 scope.go:122] "RemoveContainer" containerID="745fa541097ce9c6aee6ca7b53a3184ef7043bd6a36f7afa6e5ee0565c2f4404" Oct 30 02:15:19.532678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:19.532578 5998 scope.go:122] "RemoveContainer" containerID="d96a1eb3afa8d37942552a6657fc807e568118b6eb4c7b583eb1331c76aeacd4" Oct 30 02:15:19.532678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:19.532655 5998 scope.go:122] "RemoveContainer" containerID="ea459951f930b24e11e61647e82ca5ae0975a4410418764ead4cc7d994554962" Oct 30 02:15:20.024608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.024575 5998 generic.go:356] "Generic (PLEG): container finished" podID="a93eb477-297b-4060-a79a-8ba36de67e83" containerID="c71de2e5c8023e1ca91477e1eebb7023e25f01a420a34bcf9340bee26068339b" exitCode=60 Oct 30 02:15:20.025129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.024654 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" event={"ID":"a93eb477-297b-4060-a79a-8ba36de67e83","Type":"ContainerDied","Data":"c71de2e5c8023e1ca91477e1eebb7023e25f01a420a34bcf9340bee26068339b"} Oct 30 02:15:20.025129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.024703 5998 scope.go:122] "RemoveContainer" containerID="5e746fcfb42b8f50700442a24860f6ca573d5e0ac79beea3d9a47505707c7a45" Oct 30 02:15:20.025254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.025163 5998 scope.go:122] "RemoveContainer" containerID="c71de2e5c8023e1ca91477e1eebb7023e25f01a420a34bcf9340bee26068339b" Oct 30 02:15:20.025502 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:20.025480 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2_odh-ai-gateway-infra(a93eb477-297b-4060-a79a-8ba36de67e83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" podUID="a93eb477-297b-4060-a79a-8ba36de67e83" Oct 30 02:15:20.031851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.031819 5998 generic.go:356] "Generic (PLEG): container finished" podID="0a25d75a-85d3-4f3f-a951-aa39f35481fb" containerID="6264cc547f5b40a2fa2c3114553949a7ce4c555d63c6dddc0be15a42300385fa" exitCode=60 Oct 30 02:15:20.031982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.031916 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" event={"ID":"0a25d75a-85d3-4f3f-a951-aa39f35481fb","Type":"ContainerDied","Data":"6264cc547f5b40a2fa2c3114553949a7ce4c555d63c6dddc0be15a42300385fa"} Oct 30 02:15:20.032457 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.032434 5998 scope.go:122] "RemoveContainer" containerID="6264cc547f5b40a2fa2c3114553949a7ce4c555d63c6dddc0be15a42300385fa" Oct 30 02:15:20.032870 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:20.032847 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw_odh-ai-gateway-infra(0a25d75a-85d3-4f3f-a951-aa39f35481fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" podUID="0a25d75a-85d3-4f3f-a951-aa39f35481fb" Oct 30 02:15:20.035140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.035104 5998 generic.go:356] "Generic (PLEG): container finished" podID="7a692aea-70f7-4f61-adc6-bba42c9cc1bd" containerID="c9b7c0c3fb83a2097eca1b5ecdfc58aaf801bb0cb83f21bee145e9d2d2e92c40" exitCode=60 Oct 30 02:15:20.035252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.035189 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" event={"ID":"7a692aea-70f7-4f61-adc6-bba42c9cc1bd","Type":"ContainerDied","Data":"c9b7c0c3fb83a2097eca1b5ecdfc58aaf801bb0cb83f21bee145e9d2d2e92c40"} Oct 30 02:15:20.035676 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.035655 5998 scope.go:122] "RemoveContainer" containerID="c9b7c0c3fb83a2097eca1b5ecdfc58aaf801bb0cb83f21bee145e9d2d2e92c40" Oct 30 02:15:20.036051 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:20.036029 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc_odh-ai-gateway-infra(7a692aea-70f7-4f61-adc6-bba42c9cc1bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" podUID="7a692aea-70f7-4f61-adc6-bba42c9cc1bd" Oct 30 02:15:20.037894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.037871 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" event={"ID":"f70185f3-a978-4c11-ab66-903cfcba41b8","Type":"ContainerStarted","Data":"e73b519d5866a83f537b5b2b9474ec35869db2e8e8dff3093d47e67a61c189bb"} Oct 30 02:15:20.040243 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.040216 5998 generic.go:356] "Generic (PLEG): container finished" podID="3212debe-7561-4fa1-bdd9-bd34e88389ea" containerID="5a67f7088f613908e187fb6b4c6d395306456b1072fc8318a1f1eb5505f6321c" exitCode=60 Oct 30 02:15:20.040353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.040324 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" event={"ID":"3212debe-7561-4fa1-bdd9-bd34e88389ea","Type":"ContainerDied","Data":"5a67f7088f613908e187fb6b4c6d395306456b1072fc8318a1f1eb5505f6321c"} Oct 30 02:15:20.040734 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.040711 5998 scope.go:122] "RemoveContainer" containerID="5a67f7088f613908e187fb6b4c6d395306456b1072fc8318a1f1eb5505f6321c" Oct 30 02:15:20.041066 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:20.041040 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s_odh-ai-gateway-infra(3212debe-7561-4fa1-bdd9-bd34e88389ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" podUID="3212debe-7561-4fa1-bdd9-bd34e88389ea" Oct 30 02:15:20.043427 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.043404 5998 generic.go:356] "Generic (PLEG): container finished" podID="88d6d3ad-5fa9-4aca-9803-a981213c42af" containerID="97b4680e058afcfd1699642ee1478ea32397be32de101d70fe900061a8113de9" exitCode=60 Oct 30 02:15:20.043529 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.043476 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" event={"ID":"88d6d3ad-5fa9-4aca-9803-a981213c42af","Type":"ContainerDied","Data":"97b4680e058afcfd1699642ee1478ea32397be32de101d70fe900061a8113de9"} Oct 30 02:15:20.043917 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.043886 5998 scope.go:122] "RemoveContainer" containerID="97b4680e058afcfd1699642ee1478ea32397be32de101d70fe900061a8113de9" Oct 30 02:15:20.044239 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:20.044215 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb_odh-ai-gateway-infra(88d6d3ad-5fa9-4aca-9803-a981213c42af)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" podUID="88d6d3ad-5fa9-4aca-9803-a981213c42af" Oct 30 02:15:20.058783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.058752 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" event={"ID":"bf51a9f7-5514-42fd-9122-75d1495cde92","Type":"ContainerStarted","Data":"5db8da73e5ca5e083ed30d4cd8dd0b9602a9d177eea3b8784379276459768696"} Oct 30 02:15:20.061756 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.061733 5998 generic.go:356] "Generic (PLEG): container finished" podID="30276c96-e1d6-4759-9368-51b819812537" containerID="6923bb27bed27fa920336735c2dad3d22f7a28d97e03d8d154161236c94a42a0" exitCode=60 Oct 30 02:15:20.061859 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.061803 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" event={"ID":"30276c96-e1d6-4759-9368-51b819812537","Type":"ContainerDied","Data":"6923bb27bed27fa920336735c2dad3d22f7a28d97e03d8d154161236c94a42a0"} Oct 30 02:15:20.062148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.062134 5998 scope.go:122] "RemoveContainer" containerID="6923bb27bed27fa920336735c2dad3d22f7a28d97e03d8d154161236c94a42a0" Oct 30 02:15:20.062441 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:20.062414 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf_odh-ai-gateway-infra(30276c96-e1d6-4759-9368-51b819812537)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" podUID="30276c96-e1d6-4759-9368-51b819812537" Oct 30 02:15:20.127922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.127534 5998 scope.go:122] "RemoveContainer" containerID="9e38753c060338b49e00edd4b78e366f8114eb7a2dea6f519ea7ad6949496044" Oct 30 02:15:20.165748 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.165721 5998 scope.go:122] "RemoveContainer" containerID="312eeca8fa3280584b26b5e4d2bf53bf8eadde929380eb3c069cc77d852b40bf" Oct 30 02:15:20.184265 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.184234 5998 scope.go:122] "RemoveContainer" containerID="a5fad93a67bcc835cfbfaf58d7b1c6b2306aa700a57737df1eb612b4855491cd" Oct 30 02:15:20.232877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.205275 5998 scope.go:122] "RemoveContainer" containerID="c1f4a20b044e34a91e770eae0aa1f7b71fb54c199b2c3ea6f6d7174852507b4f" Oct 30 02:15:20.267137 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.267109 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" Oct 30 02:15:20.272478 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.272455 5998 scope.go:122] "RemoveContainer" containerID="03cb3c23ab3ca8318c0d3da6174d81e4e94919b2f930cdb2575f76609a42c9c4" Oct 30 02:15:20.318785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.318711 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" Oct 30 02:15:20.331379 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.331343 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea963033-f2cb-4e1c-a9ef-7d25c76581eb-kube-api-access-bdbzd\" (UniqueName: \"kubernetes.io/projected/ea963033-f2cb-4e1c-a9ef-7d25c76581eb-kube-api-access-bdbzd\") pod \"ea963033-f2cb-4e1c-a9ef-7d25c76581eb\" (UID: \"ea963033-f2cb-4e1c-a9ef-7d25c76581eb\") " Oct 30 02:15:20.331572 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.331410 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea963033-f2cb-4e1c-a9ef-7d25c76581eb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea963033-f2cb-4e1c-a9ef-7d25c76581eb-maas-api-service-ca\") pod \"ea963033-f2cb-4e1c-a9ef-7d25c76581eb\" (UID: \"ea963033-f2cb-4e1c-a9ef-7d25c76581eb\") " Oct 30 02:15:20.332085 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.332053 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea963033-f2cb-4e1c-a9ef-7d25c76581eb-maas-api-service-ca" pod "ea963033-f2cb-4e1c-a9ef-7d25c76581eb" (UID: "ea963033-f2cb-4e1c-a9ef-7d25c76581eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:20.334013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.333987 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea963033-f2cb-4e1c-a9ef-7d25c76581eb-kube-api-access-bdbzd" pod "ea963033-f2cb-4e1c-a9ef-7d25c76581eb" (UID: "ea963033-f2cb-4e1c-a9ef-7d25c76581eb"). InnerVolumeSpecName "kube-api-access-bdbzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:20.392011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.391975 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:20.453692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.453651 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e5a7ed-87b9-4be7-87fe-39944b774797-maas-api-service-ca" pod "54e5a7ed-87b9-4be7-87fe-39944b774797" (UID: "54e5a7ed-87b9-4be7-87fe-39944b774797"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:20.453977 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.453940 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e5a7ed-87b9-4be7-87fe-39944b774797-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e5a7ed-87b9-4be7-87fe-39944b774797-maas-api-service-ca\") pod \"54e5a7ed-87b9-4be7-87fe-39944b774797\" (UID: \"54e5a7ed-87b9-4be7-87fe-39944b774797\") " Oct 30 02:15:20.457436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.454329 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e5a7ed-87b9-4be7-87fe-39944b774797-kube-api-access-6xfkl\" (UniqueName: \"kubernetes.io/projected/54e5a7ed-87b9-4be7-87fe-39944b774797-kube-api-access-6xfkl\") pod \"54e5a7ed-87b9-4be7-87fe-39944b774797\" (UID: \"54e5a7ed-87b9-4be7-87fe-39944b774797\") " Oct 30 02:15:20.457436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.456068 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e5a7ed-87b9-4be7-87fe-39944b774797-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:20.457436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.456113 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdbzd\" (UniqueName: \"kubernetes.io/projected/ea963033-f2cb-4e1c-a9ef-7d25c76581eb-kube-api-access-bdbzd\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:20.457436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.456129 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea963033-f2cb-4e1c-a9ef-7d25c76581eb-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:20.462955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.462918 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e5a7ed-87b9-4be7-87fe-39944b774797-kube-api-access-6xfkl" pod "54e5a7ed-87b9-4be7-87fe-39944b774797" (UID: "54e5a7ed-87b9-4be7-87fe-39944b774797"). InnerVolumeSpecName "kube-api-access-6xfkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:20.562304 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:20.556992 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xfkl\" (UniqueName: \"kubernetes.io/projected/54e5a7ed-87b9-4be7-87fe-39944b774797-kube-api-access-6xfkl\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.077081 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.077043 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2"] Oct 30 02:15:21.077997 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.077965 5998 generic.go:356] "Generic (PLEG): container finished" podID="ab563a5a-c1f8-4e32-aa5d-91effabc8a18" containerID="4e1efcc82ad6c06df7fe04ecd19f51357f758fa91dde41e4046a059b4a66a443" exitCode=60 Oct 30 02:15:21.078167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.078038 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" event={"ID":"ab563a5a-c1f8-4e32-aa5d-91effabc8a18","Type":"ContainerDied","Data":"4e1efcc82ad6c06df7fe04ecd19f51357f758fa91dde41e4046a059b4a66a443"} Oct 30 02:15:21.078167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.078126 5998 scope.go:122] "RemoveContainer" containerID="ea459951f930b24e11e61647e82ca5ae0975a4410418764ead4cc7d994554962" Oct 30 02:15:21.078557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.078536 5998 scope.go:122] "RemoveContainer" containerID="4e1efcc82ad6c06df7fe04ecd19f51357f758fa91dde41e4046a059b4a66a443" Oct 30 02:15:21.078968 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:21.078932 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr_odh-ai-gateway-infra(ab563a5a-c1f8-4e32-aa5d-91effabc8a18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" podUID="ab563a5a-c1f8-4e32-aa5d-91effabc8a18" Oct 30 02:15:21.082321 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.082304 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" Oct 30 02:15:21.082425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.082329 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t" event={"ID":"ea963033-f2cb-4e1c-a9ef-7d25c76581eb","Type":"ContainerDied","Data":"2351acb341582aef755653526338c1e2cdbc2f4e157f96d67a0b15a7039199e2"} Oct 30 02:15:21.089071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.089042 5998 generic.go:356] "Generic (PLEG): container finished" podID="f70185f3-a978-4c11-ab66-903cfcba41b8" containerID="e73b519d5866a83f537b5b2b9474ec35869db2e8e8dff3093d47e67a61c189bb" exitCode=60 Oct 30 02:15:21.089223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.089142 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" event={"ID":"f70185f3-a978-4c11-ab66-903cfcba41b8","Type":"ContainerDied","Data":"e73b519d5866a83f537b5b2b9474ec35869db2e8e8dff3093d47e67a61c189bb"} Oct 30 02:15:21.089674 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.089652 5998 scope.go:122] "RemoveContainer" containerID="e73b519d5866a83f537b5b2b9474ec35869db2e8e8dff3093d47e67a61c189bb" Oct 30 02:15:21.090499 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:21.090042 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s_odh-ai-gateway-infra(f70185f3-a978-4c11-ab66-903cfcba41b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" podUID="f70185f3-a978-4c11-ab66-903cfcba41b8" Oct 30 02:15:21.093977 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.093961 5998 scope.go:122] "RemoveContainer" containerID="a97fb8e717c145ea90c2760afcfee37f7e8a832b710a731fe0ffa7458d297381" Oct 30 02:15:21.102117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.098074 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" event={"ID":"54e5a7ed-87b9-4be7-87fe-39944b774797","Type":"ContainerDied","Data":"f41d0da0a098716b80a2f3410ce55dc1bb9c88281b3d6abea58538bc4d06883b"} Oct 30 02:15:21.102117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.098078 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms" Oct 30 02:15:21.106105 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.103696 5998 generic.go:356] "Generic (PLEG): container finished" podID="bf51a9f7-5514-42fd-9122-75d1495cde92" containerID="5db8da73e5ca5e083ed30d4cd8dd0b9602a9d177eea3b8784379276459768696" exitCode=60 Oct 30 02:15:21.106105 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.103726 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" event={"ID":"bf51a9f7-5514-42fd-9122-75d1495cde92","Type":"ContainerDied","Data":"5db8da73e5ca5e083ed30d4cd8dd0b9602a9d177eea3b8784379276459768696"} Oct 30 02:15:21.106105 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.104353 5998 scope.go:122] "RemoveContainer" containerID="5db8da73e5ca5e083ed30d4cd8dd0b9602a9d177eea3b8784379276459768696" Oct 30 02:15:21.106105 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:21.104855 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb_odh-ai-gateway-infra(bf51a9f7-5514-42fd-9122-75d1495cde92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" podUID="bf51a9f7-5514-42fd-9122-75d1495cde92" Oct 30 02:15:21.109209 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.109193 5998 scope.go:122] "RemoveContainer" containerID="745fa541097ce9c6aee6ca7b53a3184ef7043bd6a36f7afa6e5ee0565c2f4404" Oct 30 02:15:21.129177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.129139 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc"] Oct 30 02:15:21.131768 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.130994 5998 scope.go:122] "RemoveContainer" containerID="ee603f10c2aaefe0ac834ca58bda70fb1e414d7defc830a288f8b26591e3fede" Oct 30 02:15:21.158960 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.158932 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t"] Oct 30 02:15:21.162412 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.162393 5998 scope.go:122] "RemoveContainer" containerID="d96a1eb3afa8d37942552a6657fc807e568118b6eb4c7b583eb1331c76aeacd4" Oct 30 02:15:21.169499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.168110 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s"] Oct 30 02:15:21.171599 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.171565 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888775-xz57t"] Oct 30 02:15:21.277227 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.277177 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf"] Oct 30 02:15:21.368335 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.361615 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms"] Oct 30 02:15:21.368587 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.368567 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9a1cd0-29888775-9l6ms"] Oct 30 02:15:21.377142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.376339 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb"] Oct 30 02:15:21.389707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.389451 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" Oct 30 02:15:21.417687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.416601 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s"] Oct 30 02:15:21.424875 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.424169 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" Oct 30 02:15:21.460396 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.460364 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw"] Oct 30 02:15:21.480560 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.480073 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb"] Oct 30 02:15:21.484053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.484020 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93eb477-297b-4060-a79a-8ba36de67e83-kube-api-access-pk42p\" (UniqueName: \"kubernetes.io/projected/a93eb477-297b-4060-a79a-8ba36de67e83-kube-api-access-pk42p\") pod \"a93eb477-297b-4060-a79a-8ba36de67e83\" (UID: \"a93eb477-297b-4060-a79a-8ba36de67e83\") " Oct 30 02:15:21.484192 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.484141 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a93eb477-297b-4060-a79a-8ba36de67e83-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a93eb477-297b-4060-a79a-8ba36de67e83-maas-api-service-ca\") pod \"a93eb477-297b-4060-a79a-8ba36de67e83\" (UID: \"a93eb477-297b-4060-a79a-8ba36de67e83\") " Oct 30 02:15:21.484913 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.484883 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93eb477-297b-4060-a79a-8ba36de67e83-maas-api-service-ca" pod "a93eb477-297b-4060-a79a-8ba36de67e83" (UID: "a93eb477-297b-4060-a79a-8ba36de67e83"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:21.485212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.485188 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a93eb477-297b-4060-a79a-8ba36de67e83-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.506608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.506578 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93eb477-297b-4060-a79a-8ba36de67e83-kube-api-access-pk42p" pod "a93eb477-297b-4060-a79a-8ba36de67e83" (UID: "a93eb477-297b-4060-a79a-8ba36de67e83"). InnerVolumeSpecName "kube-api-access-pk42p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:21.537060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.536378 5998 scope.go:122] "RemoveContainer" containerID="9094fc14d31491d82ba094f4bcfc217786beadf16f6baaa3f80552a1603d59d1" Oct 30 02:15:21.562922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.562707 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" Oct 30 02:15:21.592896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.592866 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a692aea-70f7-4f61-adc6-bba42c9cc1bd-kube-api-access-8rr2m\" (UniqueName: \"kubernetes.io/projected/7a692aea-70f7-4f61-adc6-bba42c9cc1bd-kube-api-access-8rr2m\") pod \"7a692aea-70f7-4f61-adc6-bba42c9cc1bd\" (UID: \"7a692aea-70f7-4f61-adc6-bba42c9cc1bd\") " Oct 30 02:15:21.593231 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.593210 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a692aea-70f7-4f61-adc6-bba42c9cc1bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a692aea-70f7-4f61-adc6-bba42c9cc1bd-maas-api-service-ca\") pod \"7a692aea-70f7-4f61-adc6-bba42c9cc1bd\" (UID: \"7a692aea-70f7-4f61-adc6-bba42c9cc1bd\") " Oct 30 02:15:21.594276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.594240 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk42p\" (UniqueName: \"kubernetes.io/projected/a93eb477-297b-4060-a79a-8ba36de67e83-kube-api-access-pk42p\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.594810 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.594776 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a692aea-70f7-4f61-adc6-bba42c9cc1bd-maas-api-service-ca" pod "7a692aea-70f7-4f61-adc6-bba42c9cc1bd" (UID: "7a692aea-70f7-4f61-adc6-bba42c9cc1bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:21.609160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.606377 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a692aea-70f7-4f61-adc6-bba42c9cc1bd-kube-api-access-8rr2m" pod "7a692aea-70f7-4f61-adc6-bba42c9cc1bd" (UID: "7a692aea-70f7-4f61-adc6-bba42c9cc1bd"). InnerVolumeSpecName "kube-api-access-8rr2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:21.695320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.695284 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30276c96-e1d6-4759-9368-51b819812537-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30276c96-e1d6-4759-9368-51b819812537-maas-api-service-ca\") pod \"30276c96-e1d6-4759-9368-51b819812537\" (UID: \"30276c96-e1d6-4759-9368-51b819812537\") " Oct 30 02:15:21.695906 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.695467 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30276c96-e1d6-4759-9368-51b819812537-kube-api-access-9vqct\" (UniqueName: \"kubernetes.io/projected/30276c96-e1d6-4759-9368-51b819812537-kube-api-access-9vqct\") pod \"30276c96-e1d6-4759-9368-51b819812537\" (UID: \"30276c96-e1d6-4759-9368-51b819812537\") " Oct 30 02:15:21.695906 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.695627 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30276c96-e1d6-4759-9368-51b819812537-maas-api-service-ca" pod "30276c96-e1d6-4759-9368-51b819812537" (UID: "30276c96-e1d6-4759-9368-51b819812537"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:21.696162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.696135 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rr2m\" (UniqueName: \"kubernetes.io/projected/7a692aea-70f7-4f61-adc6-bba42c9cc1bd-kube-api-access-8rr2m\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.696250 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.696163 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a692aea-70f7-4f61-adc6-bba42c9cc1bd-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.696250 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.696177 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30276c96-e1d6-4759-9368-51b819812537-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.725728 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.724639 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30276c96-e1d6-4759-9368-51b819812537-kube-api-access-9vqct" pod "30276c96-e1d6-4759-9368-51b819812537" (UID: "30276c96-e1d6-4759-9368-51b819812537"). InnerVolumeSpecName "kube-api-access-9vqct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:21.775285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.775250 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" Oct 30 02:15:21.778769 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.778741 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" Oct 30 02:15:21.783232 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.783212 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" Oct 30 02:15:21.797050 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.797018 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vqct\" (UniqueName: \"kubernetes.io/projected/30276c96-e1d6-4759-9368-51b819812537-kube-api-access-9vqct\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.898249 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.898161 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d6d3ad-5fa9-4aca-9803-a981213c42af-kube-api-access-mhqh8\" (UniqueName: \"kubernetes.io/projected/88d6d3ad-5fa9-4aca-9803-a981213c42af-kube-api-access-mhqh8\") pod \"88d6d3ad-5fa9-4aca-9803-a981213c42af\" (UID: \"88d6d3ad-5fa9-4aca-9803-a981213c42af\") " Oct 30 02:15:21.898419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.898267 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3212debe-7561-4fa1-bdd9-bd34e88389ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3212debe-7561-4fa1-bdd9-bd34e88389ea-maas-api-service-ca\") pod \"3212debe-7561-4fa1-bdd9-bd34e88389ea\" (UID: \"3212debe-7561-4fa1-bdd9-bd34e88389ea\") " Oct 30 02:15:21.898419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.898302 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a25d75a-85d3-4f3f-a951-aa39f35481fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a25d75a-85d3-4f3f-a951-aa39f35481fb-maas-api-service-ca\") pod \"0a25d75a-85d3-4f3f-a951-aa39f35481fb\" (UID: \"0a25d75a-85d3-4f3f-a951-aa39f35481fb\") " Oct 30 02:15:21.898419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.898339 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3212debe-7561-4fa1-bdd9-bd34e88389ea-kube-api-access-ndfs8\" (UniqueName: \"kubernetes.io/projected/3212debe-7561-4fa1-bdd9-bd34e88389ea-kube-api-access-ndfs8\") pod \"3212debe-7561-4fa1-bdd9-bd34e88389ea\" (UID: \"3212debe-7561-4fa1-bdd9-bd34e88389ea\") " Oct 30 02:15:21.898419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.898368 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a25d75a-85d3-4f3f-a951-aa39f35481fb-kube-api-access-pm7c9\" (UniqueName: \"kubernetes.io/projected/0a25d75a-85d3-4f3f-a951-aa39f35481fb-kube-api-access-pm7c9\") pod \"0a25d75a-85d3-4f3f-a951-aa39f35481fb\" (UID: \"0a25d75a-85d3-4f3f-a951-aa39f35481fb\") " Oct 30 02:15:21.898419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.898411 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88d6d3ad-5fa9-4aca-9803-a981213c42af-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d6d3ad-5fa9-4aca-9803-a981213c42af-maas-api-service-ca\") pod \"88d6d3ad-5fa9-4aca-9803-a981213c42af\" (UID: \"88d6d3ad-5fa9-4aca-9803-a981213c42af\") " Oct 30 02:15:21.898969 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.898940 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d6d3ad-5fa9-4aca-9803-a981213c42af-maas-api-service-ca" pod "88d6d3ad-5fa9-4aca-9803-a981213c42af" (UID: "88d6d3ad-5fa9-4aca-9803-a981213c42af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:21.899148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.899071 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a25d75a-85d3-4f3f-a951-aa39f35481fb-maas-api-service-ca" pod "0a25d75a-85d3-4f3f-a951-aa39f35481fb" (UID: "0a25d75a-85d3-4f3f-a951-aa39f35481fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:21.899923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.899890 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a25d75a-85d3-4f3f-a951-aa39f35481fb-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.900076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.899932 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d6d3ad-5fa9-4aca-9803-a981213c42af-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.900076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.899902 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3212debe-7561-4fa1-bdd9-bd34e88389ea-maas-api-service-ca" pod "3212debe-7561-4fa1-bdd9-bd34e88389ea" (UID: "3212debe-7561-4fa1-bdd9-bd34e88389ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:21.902781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.902752 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3212debe-7561-4fa1-bdd9-bd34e88389ea-kube-api-access-ndfs8" pod "3212debe-7561-4fa1-bdd9-bd34e88389ea" (UID: "3212debe-7561-4fa1-bdd9-bd34e88389ea"). InnerVolumeSpecName "kube-api-access-ndfs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:21.903544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.903457 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d6d3ad-5fa9-4aca-9803-a981213c42af-kube-api-access-mhqh8" pod "88d6d3ad-5fa9-4aca-9803-a981213c42af" (UID: "88d6d3ad-5fa9-4aca-9803-a981213c42af"). InnerVolumeSpecName "kube-api-access-mhqh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:21.903544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:21.903506 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a25d75a-85d3-4f3f-a951-aa39f35481fb-kube-api-access-pm7c9" pod "0a25d75a-85d3-4f3f-a951-aa39f35481fb" (UID: "0a25d75a-85d3-4f3f-a951-aa39f35481fb"). InnerVolumeSpecName "kube-api-access-pm7c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:22.001786 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.001755 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhqh8\" (UniqueName: \"kubernetes.io/projected/88d6d3ad-5fa9-4aca-9803-a981213c42af-kube-api-access-mhqh8\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.001786 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.001781 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3212debe-7561-4fa1-bdd9-bd34e88389ea-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.001786 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.001792 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndfs8\" (UniqueName: \"kubernetes.io/projected/3212debe-7561-4fa1-bdd9-bd34e88389ea-kube-api-access-ndfs8\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.002124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.001808 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm7c9\" (UniqueName: \"kubernetes.io/projected/0a25d75a-85d3-4f3f-a951-aa39f35481fb-kube-api-access-pm7c9\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.017274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.017236 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:22.115445 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.115414 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" Oct 30 02:15:22.115445 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.115428 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2" event={"ID":"a93eb477-297b-4060-a79a-8ba36de67e83","Type":"ContainerDied","Data":"dfbb2c649e273688da3094f1c51c2df1e76d7757f7e36915b986958448e27afa"} Oct 30 02:15:22.115987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.115502 5998 scope.go:122] "RemoveContainer" containerID="c71de2e5c8023e1ca91477e1eebb7023e25f01a420a34bcf9340bee26068339b" Oct 30 02:15:22.120830 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.120805 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" Oct 30 02:15:22.121124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.120802 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw" event={"ID":"0a25d75a-85d3-4f3f-a951-aa39f35481fb","Type":"ContainerDied","Data":"f26c0fe08a8ee15db61afc96f4e73f20764a61717f1a079ce14e3231ad632d2a"} Oct 30 02:15:22.122644 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.122625 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" Oct 30 02:15:22.123312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.123030 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc" event={"ID":"7a692aea-70f7-4f61-adc6-bba42c9cc1bd","Type":"ContainerDied","Data":"2bd3a609f230fc6f3aa7a45382e1becadeec9dfe00b22dee6764d1e7b34fd273"} Oct 30 02:15:22.131184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.128048 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" event={"ID":"3212debe-7561-4fa1-bdd9-bd34e88389ea","Type":"ContainerDied","Data":"72d7ca6eb883a0e7c75f6b61290fff9c651caf99a2568eb67972906eb8e420cf"} Oct 30 02:15:22.131184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.128168 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s" Oct 30 02:15:22.138281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.138252 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" event={"ID":"88d6d3ad-5fa9-4aca-9803-a981213c42af","Type":"ContainerDied","Data":"d57bfdfe84268db398581bf0449b96e5d67f6a2ff91927b417c0be2a1c3c102f"} Oct 30 02:15:22.138375 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.138359 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb" Oct 30 02:15:22.140746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.140724 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr"] Oct 30 02:15:22.150324 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.149416 5998 scope.go:122] "RemoveContainer" containerID="6264cc547f5b40a2fa2c3114553949a7ce4c555d63c6dddc0be15a42300385fa" Oct 30 02:15:22.150324 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.149650 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" Oct 30 02:15:22.150324 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.149643 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf" event={"ID":"30276c96-e1d6-4759-9368-51b819812537","Type":"ContainerDied","Data":"474349e3c9a122ab19726eb0af82ebb327a0f91bef2f309bb890483afc543fe6"} Oct 30 02:15:22.160889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.160861 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" event={"ID":"287b0a33-1dc7-4284-8c93-7af661cf9d37","Type":"ContainerStarted","Data":"2992e424a183c31eb57a05e583f64c94d230b9b2d5dfe0bebd1acf4251e70278"} Oct 30 02:15:22.170861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.170842 5998 scope.go:122] "RemoveContainer" containerID="c9b7c0c3fb83a2097eca1b5ecdfc58aaf801bb0cb83f21bee145e9d2d2e92c40" Oct 30 02:15:22.190502 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.190379 5998 scope.go:122] "RemoveContainer" containerID="5a67f7088f613908e187fb6b4c6d395306456b1072fc8318a1f1eb5505f6321c" Oct 30 02:15:22.204611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.204495 5998 scope.go:122] "RemoveContainer" containerID="97b4680e058afcfd1699642ee1478ea32397be32de101d70fe900061a8113de9" Oct 30 02:15:22.237643 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.229652 5998 scope.go:122] "RemoveContainer" containerID="6923bb27bed27fa920336735c2dad3d22f7a28d97e03d8d154161236c94a42a0" Oct 30 02:15:22.278031 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.278000 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2"] Oct 30 02:15:22.281986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.281959 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-2819d2-29888775-x42d2"] Oct 30 02:15:22.312447 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.312416 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s"] Oct 30 02:15:22.316915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.316883 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888775-dlk8s"] Oct 30 02:15:22.333456 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.333417 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" Oct 30 02:15:22.411188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.406888 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f70185f3-a978-4c11-ab66-903cfcba41b8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70185f3-a978-4c11-ab66-903cfcba41b8-maas-api-service-ca\") pod \"f70185f3-a978-4c11-ab66-903cfcba41b8\" (UID: \"f70185f3-a978-4c11-ab66-903cfcba41b8\") " Oct 30 02:15:22.411188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.407065 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70185f3-a978-4c11-ab66-903cfcba41b8-kube-api-access-nxnbh\" (UniqueName: \"kubernetes.io/projected/f70185f3-a978-4c11-ab66-903cfcba41b8-kube-api-access-nxnbh\") pod \"f70185f3-a978-4c11-ab66-903cfcba41b8\" (UID: \"f70185f3-a978-4c11-ab66-903cfcba41b8\") " Oct 30 02:15:22.411188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.410334 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70185f3-a978-4c11-ab66-903cfcba41b8-maas-api-service-ca" pod "f70185f3-a978-4c11-ab66-903cfcba41b8" (UID: "f70185f3-a978-4c11-ab66-903cfcba41b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:22.412461 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.412427 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70185f3-a978-4c11-ab66-903cfcba41b8-kube-api-access-nxnbh" pod "f70185f3-a978-4c11-ab66-903cfcba41b8" (UID: "f70185f3-a978-4c11-ab66-903cfcba41b8"). InnerVolumeSpecName "kube-api-access-nxnbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:22.428311 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.428278 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" Oct 30 02:15:22.442017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.440808 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw"] Oct 30 02:15:22.445896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.445867 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888775-q5glw"] Oct 30 02:15:22.449624 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.449597 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" Oct 30 02:15:22.468483 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.468453 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb"] Oct 30 02:15:22.479377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.479348 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888775-6nqcb"] Oct 30 02:15:22.510325 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.510287 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf51a9f7-5514-42fd-9122-75d1495cde92-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf51a9f7-5514-42fd-9122-75d1495cde92-maas-api-service-ca\") pod \"bf51a9f7-5514-42fd-9122-75d1495cde92\" (UID: \"bf51a9f7-5514-42fd-9122-75d1495cde92\") " Oct 30 02:15:22.510506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.510485 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf51a9f7-5514-42fd-9122-75d1495cde92-kube-api-access-d97zr\" (UniqueName: \"kubernetes.io/projected/bf51a9f7-5514-42fd-9122-75d1495cde92-kube-api-access-d97zr\") pod \"bf51a9f7-5514-42fd-9122-75d1495cde92\" (UID: \"bf51a9f7-5514-42fd-9122-75d1495cde92\") " Oct 30 02:15:22.511671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.511595 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf51a9f7-5514-42fd-9122-75d1495cde92-maas-api-service-ca" pod "bf51a9f7-5514-42fd-9122-75d1495cde92" (UID: "bf51a9f7-5514-42fd-9122-75d1495cde92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:22.511671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.511079 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f70185f3-a978-4c11-ab66-903cfcba41b8-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.511671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.511662 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxnbh\" (UniqueName: \"kubernetes.io/projected/f70185f3-a978-4c11-ab66-903cfcba41b8-kube-api-access-nxnbh\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.513864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.513829 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc"] Oct 30 02:15:22.514479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.514449 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf51a9f7-5514-42fd-9122-75d1495cde92-kube-api-access-d97zr" pod "bf51a9f7-5514-42fd-9122-75d1495cde92" (UID: "bf51a9f7-5514-42fd-9122-75d1495cde92"). InnerVolumeSpecName "kube-api-access-d97zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:22.518394 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.518372 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a0684f-29888775-2srwc"] Oct 30 02:15:22.532172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.532151 5998 scope.go:122] "RemoveContainer" containerID="21c065a8ec10db0b25e3e35a529a7210cefe9874cc1c7b6ea5e8f0b804c47e58" Oct 30 02:15:22.543777 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.543744 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a25d75a-85d3-4f3f-a951-aa39f35481fb" path="/var/lib/kubelet/pods/0a25d75a-85d3-4f3f-a951-aa39f35481fb/volumes" Oct 30 02:15:22.544402 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.544381 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3212debe-7561-4fa1-bdd9-bd34e88389ea" path="/var/lib/kubelet/pods/3212debe-7561-4fa1-bdd9-bd34e88389ea/volumes" Oct 30 02:15:22.544936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.544917 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e5a7ed-87b9-4be7-87fe-39944b774797" path="/var/lib/kubelet/pods/54e5a7ed-87b9-4be7-87fe-39944b774797/volumes" Oct 30 02:15:22.545900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.545878 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a692aea-70f7-4f61-adc6-bba42c9cc1bd" path="/var/lib/kubelet/pods/7a692aea-70f7-4f61-adc6-bba42c9cc1bd/volumes" Oct 30 02:15:22.546599 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.546576 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88d6d3ad-5fa9-4aca-9803-a981213c42af" path="/var/lib/kubelet/pods/88d6d3ad-5fa9-4aca-9803-a981213c42af/volumes" Oct 30 02:15:22.547145 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.547115 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a93eb477-297b-4060-a79a-8ba36de67e83" path="/var/lib/kubelet/pods/a93eb477-297b-4060-a79a-8ba36de67e83/volumes" Oct 30 02:15:22.548013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.547993 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea963033-f2cb-4e1c-a9ef-7d25c76581eb" path="/var/lib/kubelet/pods/ea963033-f2cb-4e1c-a9ef-7d25c76581eb/volumes" Oct 30 02:15:22.555622 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.555598 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf"] Oct 30 02:15:22.555794 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.555773 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888775-h8qzf"] Oct 30 02:15:22.612428 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.612394 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab563a5a-c1f8-4e32-aa5d-91effabc8a18-kube-api-access-v84gw\" (UniqueName: \"kubernetes.io/projected/ab563a5a-c1f8-4e32-aa5d-91effabc8a18-kube-api-access-v84gw\") pod \"ab563a5a-c1f8-4e32-aa5d-91effabc8a18\" (UID: \"ab563a5a-c1f8-4e32-aa5d-91effabc8a18\") " Oct 30 02:15:22.612584 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.612437 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab563a5a-c1f8-4e32-aa5d-91effabc8a18-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab563a5a-c1f8-4e32-aa5d-91effabc8a18-maas-api-service-ca\") pod \"ab563a5a-c1f8-4e32-aa5d-91effabc8a18\" (UID: \"ab563a5a-c1f8-4e32-aa5d-91effabc8a18\") " Oct 30 02:15:22.612921 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.612891 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab563a5a-c1f8-4e32-aa5d-91effabc8a18-maas-api-service-ca" pod "ab563a5a-c1f8-4e32-aa5d-91effabc8a18" (UID: "ab563a5a-c1f8-4e32-aa5d-91effabc8a18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:22.613414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.613394 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d97zr\" (UniqueName: \"kubernetes.io/projected/bf51a9f7-5514-42fd-9122-75d1495cde92-kube-api-access-d97zr\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.613543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.613528 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab563a5a-c1f8-4e32-aa5d-91effabc8a18-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.613627 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.613613 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf51a9f7-5514-42fd-9122-75d1495cde92-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.615157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.615132 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab563a5a-c1f8-4e32-aa5d-91effabc8a18-kube-api-access-v84gw" pod "ab563a5a-c1f8-4e32-aa5d-91effabc8a18" (UID: "ab563a5a-c1f8-4e32-aa5d-91effabc8a18"). InnerVolumeSpecName "kube-api-access-v84gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:22.716507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:22.716390 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v84gw\" (UniqueName: \"kubernetes.io/projected/ab563a5a-c1f8-4e32-aa5d-91effabc8a18-kube-api-access-v84gw\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:23.171722 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.171691 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" Oct 30 02:15:23.172245 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.171682 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb" event={"ID":"bf51a9f7-5514-42fd-9122-75d1495cde92","Type":"ContainerDied","Data":"d3c1fa0d604936191a3e3cd80a4c5c2b771000961ac9ec521c776b2dc874ed12"} Oct 30 02:15:23.172245 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.171852 5998 scope.go:122] "RemoveContainer" containerID="5db8da73e5ca5e083ed30d4cd8dd0b9602a9d177eea3b8784379276459768696" Oct 30 02:15:23.174842 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.174803 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" event={"ID":"ab563a5a-c1f8-4e32-aa5d-91effabc8a18","Type":"ContainerDied","Data":"48042c6da38079479657b986d12da2433234507956ae08a8a48842702359f8b8"} Oct 30 02:15:23.174992 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.174883 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr" Oct 30 02:15:23.181185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.181155 5998 generic.go:356] "Generic (PLEG): container finished" podID="287b0a33-1dc7-4284-8c93-7af661cf9d37" containerID="2992e424a183c31eb57a05e583f64c94d230b9b2d5dfe0bebd1acf4251e70278" exitCode=60 Oct 30 02:15:23.181298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.181280 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" event={"ID":"287b0a33-1dc7-4284-8c93-7af661cf9d37","Type":"ContainerDied","Data":"2992e424a183c31eb57a05e583f64c94d230b9b2d5dfe0bebd1acf4251e70278"} Oct 30 02:15:23.181754 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.181734 5998 scope.go:122] "RemoveContainer" containerID="2992e424a183c31eb57a05e583f64c94d230b9b2d5dfe0bebd1acf4251e70278" Oct 30 02:15:23.182112 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:23.182070 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7_odh-ai-gateway-infra(287b0a33-1dc7-4284-8c93-7af661cf9d37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" podUID="287b0a33-1dc7-4284-8c93-7af661cf9d37" Oct 30 02:15:23.187272 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.187013 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" event={"ID":"3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4","Type":"ContainerStarted","Data":"2e71c7f9e84bf0bf377f14e5ad7491dc2abba99dbd3093a62ea2c9df6b63404e"} Oct 30 02:15:23.188386 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.187788 5998 scope.go:122] "RemoveContainer" containerID="4e1efcc82ad6c06df7fe04ecd19f51357f758fa91dde41e4046a059b4a66a443" Oct 30 02:15:23.193813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.193793 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" Oct 30 02:15:23.193922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.193796 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s" event={"ID":"f70185f3-a978-4c11-ab66-903cfcba41b8","Type":"ContainerDied","Data":"961ee16e7f9d6423f25de0ef68c0391edc7bd037662169ee712a28e935ded8de"} Oct 30 02:15:23.199343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.199320 5998 scope.go:122] "RemoveContainer" containerID="9094fc14d31491d82ba094f4bcfc217786beadf16f6baaa3f80552a1603d59d1" Oct 30 02:15:23.213994 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.213970 5998 scope.go:122] "RemoveContainer" containerID="e73b519d5866a83f537b5b2b9474ec35869db2e8e8dff3093d47e67a61c189bb" Oct 30 02:15:23.221082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.221058 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb"] Oct 30 02:15:23.226656 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.226631 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888775-df8jb"] Oct 30 02:15:23.306517 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.306481 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr"] Oct 30 02:15:23.312857 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.312829 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888775-kspmr"] Oct 30 02:15:23.383005 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.382971 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s"] Oct 30 02:15:23.386944 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.386918 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-58fad4-29888775-bpp4s"] Oct 30 02:15:23.411336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.411307 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7"] Oct 30 02:15:23.687313 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:23.687217 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:24.243646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:24.243609 5998 generic.go:356] "Generic (PLEG): container finished" podID="3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4" containerID="2e71c7f9e84bf0bf377f14e5ad7491dc2abba99dbd3093a62ea2c9df6b63404e" exitCode=60 Oct 30 02:15:24.244406 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:24.244376 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" event={"ID":"3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4","Type":"ContainerDied","Data":"2e71c7f9e84bf0bf377f14e5ad7491dc2abba99dbd3093a62ea2c9df6b63404e"} Oct 30 02:15:24.244591 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:24.244575 5998 scope.go:122] "RemoveContainer" containerID="21c065a8ec10db0b25e3e35a529a7210cefe9874cc1c7b6ea5e8f0b804c47e58" Oct 30 02:15:24.246228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:24.246192 5998 scope.go:122] "RemoveContainer" containerID="2e71c7f9e84bf0bf377f14e5ad7491dc2abba99dbd3093a62ea2c9df6b63404e" Oct 30 02:15:24.265753 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:24.265721 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr_odh-ai-gateway-infra(3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" podUID="3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4" Oct 30 02:15:24.372377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:24.367678 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr"] Oct 30 02:15:24.469406 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:24.469380 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" Oct 30 02:15:24.543343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:24.543253 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30276c96-e1d6-4759-9368-51b819812537" path="/var/lib/kubelet/pods/30276c96-e1d6-4759-9368-51b819812537/volumes" Oct 30 02:15:24.543857 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:24.543832 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab563a5a-c1f8-4e32-aa5d-91effabc8a18" path="/var/lib/kubelet/pods/ab563a5a-c1f8-4e32-aa5d-91effabc8a18/volumes" Oct 30 02:15:24.544354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:24.544333 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf51a9f7-5514-42fd-9122-75d1495cde92" path="/var/lib/kubelet/pods/bf51a9f7-5514-42fd-9122-75d1495cde92/volumes" Oct 30 02:15:24.545116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:24.545078 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f70185f3-a978-4c11-ab66-903cfcba41b8" path="/var/lib/kubelet/pods/f70185f3-a978-4c11-ab66-903cfcba41b8/volumes" Oct 30 02:15:24.659099 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:24.659053 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/287b0a33-1dc7-4284-8c93-7af661cf9d37-kube-api-access-ffvj5\" (UniqueName: \"kubernetes.io/projected/287b0a33-1dc7-4284-8c93-7af661cf9d37-kube-api-access-ffvj5\") pod \"287b0a33-1dc7-4284-8c93-7af661cf9d37\" (UID: \"287b0a33-1dc7-4284-8c93-7af661cf9d37\") " Oct 30 02:15:24.659278 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:24.659269 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/287b0a33-1dc7-4284-8c93-7af661cf9d37-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/287b0a33-1dc7-4284-8c93-7af661cf9d37-maas-api-service-ca\") pod \"287b0a33-1dc7-4284-8c93-7af661cf9d37\" (UID: \"287b0a33-1dc7-4284-8c93-7af661cf9d37\") " Oct 30 02:15:24.659618 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:24.659589 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/287b0a33-1dc7-4284-8c93-7af661cf9d37-maas-api-service-ca" pod "287b0a33-1dc7-4284-8c93-7af661cf9d37" (UID: "287b0a33-1dc7-4284-8c93-7af661cf9d37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:24.661245 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:24.661212 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287b0a33-1dc7-4284-8c93-7af661cf9d37-kube-api-access-ffvj5" pod "287b0a33-1dc7-4284-8c93-7af661cf9d37" (UID: "287b0a33-1dc7-4284-8c93-7af661cf9d37"). InnerVolumeSpecName "kube-api-access-ffvj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:24.761013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:24.760964 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/287b0a33-1dc7-4284-8c93-7af661cf9d37-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:24.761013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:24.761005 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffvj5\" (UniqueName: \"kubernetes.io/projected/287b0a33-1dc7-4284-8c93-7af661cf9d37-kube-api-access-ffvj5\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:25.223460 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:25.223418 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:15:25.327111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:25.326567 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" event={"ID":"287b0a33-1dc7-4284-8c93-7af661cf9d37","Type":"ContainerDied","Data":"0ad7c3ae07375925ab6fc87ac48919c7a915d70762e9aa1404c12c1368cc3b2c"} Oct 30 02:15:25.327111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:25.326639 5998 scope.go:122] "RemoveContainer" containerID="2992e424a183c31eb57a05e583f64c94d230b9b2d5dfe0bebd1acf4251e70278" Oct 30 02:15:25.327111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:25.326748 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7" Oct 30 02:15:25.424294 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:25.424224 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7"] Oct 30 02:15:25.432987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:25.432935 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888775-7wff7"] Oct 30 02:15:25.568904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:25.568701 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:25.598097 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:25.598060 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" Oct 30 02:15:25.707163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:25.690584 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4-maas-api-service-ca\") pod \"3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4\" (UID: \"3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4\") " Oct 30 02:15:25.707492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:25.707466 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4-kube-api-access-6hcf7\" (UniqueName: \"kubernetes.io/projected/3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4-kube-api-access-6hcf7\") pod \"3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4\" (UID: \"3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4\") " Oct 30 02:15:25.708921 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:25.691170 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4-maas-api-service-ca" pod "3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4" (UID: "3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:25.714189 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:25.713683 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4-kube-api-access-6hcf7" pod "3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4" (UID: "3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4"). InnerVolumeSpecName "kube-api-access-6hcf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:25.823176 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:25.813400 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:25.823176 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:25.813434 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hcf7\" (UniqueName: \"kubernetes.io/projected/3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4-kube-api-access-6hcf7\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:25.853112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:25.852522 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:25.867919 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:25.867143 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4"] Oct 30 02:15:26.396762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.396727 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" Oct 30 02:15:26.396762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.396747 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr" event={"ID":"3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4","Type":"ContainerDied","Data":"11581371b1bcbf349c6bf58622113f714c8c7352444ae6de58ab3c09ffa0793d"} Oct 30 02:15:26.397282 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.396810 5998 scope.go:122] "RemoveContainer" containerID="2e71c7f9e84bf0bf377f14e5ad7491dc2abba99dbd3093a62ea2c9df6b63404e" Oct 30 02:15:26.499716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.499683 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr"] Oct 30 02:15:26.520715 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.518551 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:26.541113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.532754 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888775-mpnqr"] Oct 30 02:15:26.574572 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.574530 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="287b0a33-1dc7-4284-8c93-7af661cf9d37" path="/var/lib/kubelet/pods/287b0a33-1dc7-4284-8c93-7af661cf9d37/volumes" Oct 30 02:15:26.575012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.574996 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4" path="/var/lib/kubelet/pods/3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4/volumes" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.906654 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2"] Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907645 5998 cpu_manager.go:414] "containerMap: removing container" podUID="0a25d75a-85d3-4f3f-a951-aa39f35481fb" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907668 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a25d75a-85d3-4f3f-a951-aa39f35481fb" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907682 5998 cpu_manager.go:414] "containerMap: removing container" podUID="0a25d75a-85d3-4f3f-a951-aa39f35481fb" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907691 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a25d75a-85d3-4f3f-a951-aa39f35481fb" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907703 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a93eb477-297b-4060-a79a-8ba36de67e83" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907713 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93eb477-297b-4060-a79a-8ba36de67e83" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907724 5998 cpu_manager.go:414] "containerMap: removing container" podUID="88d6d3ad-5fa9-4aca-9803-a981213c42af" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907731 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d6d3ad-5fa9-4aca-9803-a981213c42af" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907739 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3212debe-7561-4fa1-bdd9-bd34e88389ea" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907747 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3212debe-7561-4fa1-bdd9-bd34e88389ea" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907760 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3212debe-7561-4fa1-bdd9-bd34e88389ea" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907767 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3212debe-7561-4fa1-bdd9-bd34e88389ea" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907777 5998 cpu_manager.go:414] "containerMap: removing container" podUID="287b0a33-1dc7-4284-8c93-7af661cf9d37" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907783 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287b0a33-1dc7-4284-8c93-7af661cf9d37" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907792 5998 cpu_manager.go:414] "containerMap: removing container" podUID="bf51a9f7-5514-42fd-9122-75d1495cde92" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907799 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf51a9f7-5514-42fd-9122-75d1495cde92" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907807 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ab563a5a-c1f8-4e32-aa5d-91effabc8a18" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907813 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab563a5a-c1f8-4e32-aa5d-91effabc8a18" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907825 5998 cpu_manager.go:414] "containerMap: removing container" podUID="89f01e25-ada7-489b-966d-823f8bc623a1" containerName="revoke-keys" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907831 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89f01e25-ada7-489b-966d-823f8bc623a1" containerName="revoke-keys" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907847 5998 cpu_manager.go:414] "containerMap: removing container" podUID="54e5a7ed-87b9-4be7-87fe-39944b774797" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907853 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e5a7ed-87b9-4be7-87fe-39944b774797" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907863 5998 cpu_manager.go:414] "containerMap: removing container" podUID="f70185f3-a978-4c11-ab66-903cfcba41b8" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907869 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70185f3-a978-4c11-ab66-903cfcba41b8" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907882 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907889 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907898 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ea963033-f2cb-4e1c-a9ef-7d25c76581eb" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907904 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea963033-f2cb-4e1c-a9ef-7d25c76581eb" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907912 5998 cpu_manager.go:414] "containerMap: removing container" podUID="54e5a7ed-87b9-4be7-87fe-39944b774797" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907918 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e5a7ed-87b9-4be7-87fe-39944b774797" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907931 5998 cpu_manager.go:414] "containerMap: removing container" podUID="30276c96-e1d6-4759-9368-51b819812537" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907937 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30276c96-e1d6-4759-9368-51b819812537" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907947 5998 cpu_manager.go:414] "containerMap: removing container" podUID="287b0a33-1dc7-4284-8c93-7af661cf9d37" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907953 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287b0a33-1dc7-4284-8c93-7af661cf9d37" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907964 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a93eb477-297b-4060-a79a-8ba36de67e83" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907970 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93eb477-297b-4060-a79a-8ba36de67e83" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907980 5998 cpu_manager.go:414] "containerMap: removing container" podUID="bf51a9f7-5514-42fd-9122-75d1495cde92" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.907987 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf51a9f7-5514-42fd-9122-75d1495cde92" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908003 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ab563a5a-c1f8-4e32-aa5d-91effabc8a18" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908009 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab563a5a-c1f8-4e32-aa5d-91effabc8a18" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908022 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7a692aea-70f7-4f61-adc6-bba42c9cc1bd" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908028 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a692aea-70f7-4f61-adc6-bba42c9cc1bd" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908036 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7a692aea-70f7-4f61-adc6-bba42c9cc1bd" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908042 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a692aea-70f7-4f61-adc6-bba42c9cc1bd" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908050 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ea963033-f2cb-4e1c-a9ef-7d25c76581eb" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908057 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea963033-f2cb-4e1c-a9ef-7d25c76581eb" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908066 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ea963033-f2cb-4e1c-a9ef-7d25c76581eb" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908073 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea963033-f2cb-4e1c-a9ef-7d25c76581eb" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908081 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908105 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908120 5998 cpu_manager.go:414] "containerMap: removing container" podUID="54e5a7ed-87b9-4be7-87fe-39944b774797" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908126 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e5a7ed-87b9-4be7-87fe-39944b774797" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908134 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908141 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908157 5998 cpu_manager.go:414] "containerMap: removing container" podUID="bf51a9f7-5514-42fd-9122-75d1495cde92" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908165 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf51a9f7-5514-42fd-9122-75d1495cde92" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908180 5998 cpu_manager.go:414] "containerMap: removing container" podUID="30276c96-e1d6-4759-9368-51b819812537" containerName="cleanup" Oct 30 02:15:26.908132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.908187 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30276c96-e1d6-4759-9368-51b819812537" containerName="cleanup" Oct 30 02:15:27.004144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:26.999619 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:27.151840 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.151780 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" containerName="istio-proxy" Oct 30 02:15:27.164288 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.164238 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" containerName="istio-proxy" Oct 30 02:15:27.164598 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.164530 5998 cpu_manager.go:414] "containerMap: removing container" podUID="f70185f3-a978-4c11-ab66-903cfcba41b8" containerName="cleanup" Oct 30 02:15:27.164598 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.164548 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70185f3-a978-4c11-ab66-903cfcba41b8" containerName="cleanup" Oct 30 02:15:27.164829 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.164786 5998 cpu_manager.go:414] "containerMap: removing container" podUID="88d6d3ad-5fa9-4aca-9803-a981213c42af" containerName="cleanup" Oct 30 02:15:27.164829 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.164800 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d6d3ad-5fa9-4aca-9803-a981213c42af" containerName="cleanup" Oct 30 02:15:27.165561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.165470 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d6d3ad-5fa9-4aca-9803-a981213c42af" containerName="cleanup" Oct 30 02:15:27.165561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.165518 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d6d3ad-5fa9-4aca-9803-a981213c42af" containerName="cleanup" Oct 30 02:15:27.165561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.165538 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70185f3-a978-4c11-ab66-903cfcba41b8" containerName="cleanup" Oct 30 02:15:27.169276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.169170 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="287b0a33-1dc7-4284-8c93-7af661cf9d37" containerName="cleanup" Oct 30 02:15:27.169276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.169225 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a25d75a-85d3-4f3f-a951-aa39f35481fb" containerName="cleanup" Oct 30 02:15:27.169276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.169234 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a25d75a-85d3-4f3f-a951-aa39f35481fb" containerName="cleanup" Oct 30 02:15:27.169276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.169252 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93eb477-297b-4060-a79a-8ba36de67e83" containerName="cleanup" Oct 30 02:15:27.173273 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.173227 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d6d3ad-5fa9-4aca-9803-a981213c42af" containerName="cleanup" Oct 30 02:15:27.173506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.173433 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3212debe-7561-4fa1-bdd9-bd34e88389ea" containerName="cleanup" Oct 30 02:15:27.173506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.173462 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3212debe-7561-4fa1-bdd9-bd34e88389ea" containerName="cleanup" Oct 30 02:15:27.173506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.173480 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="287b0a33-1dc7-4284-8c93-7af661cf9d37" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175183 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf51a9f7-5514-42fd-9122-75d1495cde92" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175206 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab563a5a-c1f8-4e32-aa5d-91effabc8a18" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175221 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="89f01e25-ada7-489b-966d-823f8bc623a1" containerName="revoke-keys" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175237 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e5a7ed-87b9-4be7-87fe-39944b774797" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175247 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70185f3-a978-4c11-ab66-903cfcba41b8" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175272 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175282 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea963033-f2cb-4e1c-a9ef-7d25c76581eb" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175290 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e5a7ed-87b9-4be7-87fe-39944b774797" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175312 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="30276c96-e1d6-4759-9368-51b819812537" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175328 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="287b0a33-1dc7-4284-8c93-7af661cf9d37" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175345 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93eb477-297b-4060-a79a-8ba36de67e83" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175354 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf51a9f7-5514-42fd-9122-75d1495cde92" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175372 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3212debe-7561-4fa1-bdd9-bd34e88389ea" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175383 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a692aea-70f7-4f61-adc6-bba42c9cc1bd" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175398 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a692aea-70f7-4f61-adc6-bba42c9cc1bd" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175408 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea963033-f2cb-4e1c-a9ef-7d25c76581eb" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175417 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175435 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aada9e5-ef1a-4f4c-87cb-9d29fcd1f5a4" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175449 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="30276c96-e1d6-4759-9368-51b819812537" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175467 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="30276c96-e1d6-4759-9368-51b819812537" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175492 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab563a5a-c1f8-4e32-aa5d-91effabc8a18" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175503 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a692aea-70f7-4f61-adc6-bba42c9cc1bd" containerName="cleanup" Oct 30 02:15:27.182301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.175520 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0e07e1-c219-46aa-80e2-bf0eaf2ec85e" containerName="istio-proxy" Oct 30 02:15:27.471179 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.457269 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:27.586872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.586827 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:27.592943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.592913 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2"] Oct 30 02:15:27.599108 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.593541 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.625047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.624461 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-9bf0e2-openshift-default-dockercfg-sllzz\"" Oct 30 02:15:27.695810 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.695767 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-data\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.695810 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.695810 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-workload-socket\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.696108 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.695960 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5l79\" (UniqueName: \"kubernetes.io/projected/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-kube-api-access-b5l79\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.696108 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.696036 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-credential-socket\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.696108 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.696055 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-podinfo\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.696280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.696142 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istiod-ca-cert\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.696280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.696175 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-token\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.696280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.696209 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-envoy\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.696280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.696227 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-workload-certs\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.797244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.797149 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-envoy\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.797244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.797190 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-workload-certs\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.797469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.797248 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-data\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.797469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.797266 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-workload-socket\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.797469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.797331 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5l79\" (UniqueName: \"kubernetes.io/projected/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-kube-api-access-b5l79\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.797469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.797368 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-credential-socket\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.797469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.797382 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-podinfo\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.797469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.797435 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istiod-ca-cert\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.797469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.797461 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-token\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.798421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.798393 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-data\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.798528 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.798502 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-workload-certs\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.798575 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.798555 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-workload-socket\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.798803 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.798778 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-credential-socket\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.799271 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.799250 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istiod-ca-cert\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.825112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.806695 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-envoy\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.832202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.832120 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-token\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.848138 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.840133 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5l79\" (UniqueName: \"kubernetes.io/projected/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-kube-api-access-b5l79\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.848138 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.844591 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-podinfo\") pod \"e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:27.943112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:27.941353 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:28.490529 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:28.489223 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2"] Oct 30 02:15:28.491589 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:15:28.491548 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e65fd4e_e7a0_4124_88cf_c08e9f83a5ff.slice/crio-0015764ffd8a1b70445ba61034f831838c17e1f170fb4f32f5c8ce1a6148b7b2 WatchSource:0}: Error finding container 0015764ffd8a1b70445ba61034f831838c17e1f170fb4f32f5c8ce1a6148b7b2: Status 404 returned error can't find the container with id 0015764ffd8a1b70445ba61034f831838c17e1f170fb4f32f5c8ce1a6148b7b2 Oct 30 02:15:28.494845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:28.494805 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:15:28.494938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:28.494893 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:15:28.494979 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:28.494934 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:15:28.654152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:28.648404 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" event={"ID":"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff","Type":"ContainerStarted","Data":"0015764ffd8a1b70445ba61034f831838c17e1f170fb4f32f5c8ce1a6148b7b2"} Oct 30 02:15:29.020033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:29.019938 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:29.458577 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:29.458543 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:29.679872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:29.679825 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" event={"ID":"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff","Type":"ContainerStarted","Data":"1f94efb447b54eabe22a074dbc9fcb4c79542aa954598ae691ae82c5fcade00d"} Oct 30 02:15:29.951169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:29.951135 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:29.968439 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:29.968407 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:30.040801 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:30.040735 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" podStartSLOduration=4.040714812 podStartE2EDuration="4.040714812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:29.743044571 +0000 UTC m=+2759.321835654" watchObservedRunningTime="2026-10-30 02:15:30.040714812 +0000 UTC m=+2759.619505894" Oct 30 02:15:30.594255 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:30.594217 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:30.686445 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:30.686413 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:30.687643 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:30.687621 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:15:30.716246 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:30.716215 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:31.089893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:31.089842 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:31.591297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:31.591252 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:31.869779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:31.869679 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" podUID="80f35655-46ac-4a54-8326-077b1c659773" containerName="main" containerID="cri-o://5e6474eeded05740c4a7d52e4f315322151df794102e2172ab21c524016dad65" gracePeriod=45 Oct 30 02:15:31.888397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:31.888367 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:32.017881 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.012552 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:32.224956 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.224806 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:32.312868 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.312769 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:15:32.342935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.342902 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:32.372262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.372229 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-dshm\") pod \"80f35655-46ac-4a54-8326-077b1c659773\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " Oct 30 02:15:32.372417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.372294 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f35655-46ac-4a54-8326-077b1c659773-kube-api-access-lxhpn\" (UniqueName: \"kubernetes.io/projected/80f35655-46ac-4a54-8326-077b1c659773-kube-api-access-lxhpn\") pod \"80f35655-46ac-4a54-8326-077b1c659773\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " Oct 30 02:15:32.372417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.372328 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-tmp-dir\") pod \"80f35655-46ac-4a54-8326-077b1c659773\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " Oct 30 02:15:32.372559 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.372523 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-model-cache\") pod \"80f35655-46ac-4a54-8326-077b1c659773\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " Oct 30 02:15:32.372616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.372565 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-home\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-home\") pod \"80f35655-46ac-4a54-8326-077b1c659773\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " Oct 30 02:15:32.372852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.372723 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80f35655-46ac-4a54-8326-077b1c659773-tls-certs\" (UniqueName: \"kubernetes.io/secret/80f35655-46ac-4a54-8326-077b1c659773-tls-certs\") pod \"80f35655-46ac-4a54-8326-077b1c659773\" (UID: \"80f35655-46ac-4a54-8326-077b1c659773\") " Oct 30 02:15:32.373528 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.373499 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-model-cache" pod "80f35655-46ac-4a54-8326-077b1c659773" (UID: "80f35655-46ac-4a54-8326-077b1c659773"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:32.373911 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.373886 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-home" pod "80f35655-46ac-4a54-8326-077b1c659773" (UID: "80f35655-46ac-4a54-8326-077b1c659773"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:32.373988 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.373950 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-tmp-dir" pod "80f35655-46ac-4a54-8326-077b1c659773" (UID: "80f35655-46ac-4a54-8326-077b1c659773"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:32.376593 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.376463 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f35655-46ac-4a54-8326-077b1c659773-tls-certs" pod "80f35655-46ac-4a54-8326-077b1c659773" (UID: "80f35655-46ac-4a54-8326-077b1c659773"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:15:32.377162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.377129 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-dshm" pod "80f35655-46ac-4a54-8326-077b1c659773" (UID: "80f35655-46ac-4a54-8326-077b1c659773"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:32.377478 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.377448 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f35655-46ac-4a54-8326-077b1c659773-kube-api-access-lxhpn" pod "80f35655-46ac-4a54-8326-077b1c659773" (UID: "80f35655-46ac-4a54-8326-077b1c659773"). InnerVolumeSpecName "kube-api-access-lxhpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:32.474562 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.474481 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f35655-46ac-4a54-8326-077b1c659773-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:32.474562 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.474517 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:32.474562 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.474532 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxhpn\" (UniqueName: \"kubernetes.io/projected/80f35655-46ac-4a54-8326-077b1c659773-kube-api-access-lxhpn\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:32.474562 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.474549 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:32.474804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.474565 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:32.474804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.474578 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80f35655-46ac-4a54-8326-077b1c659773-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:32.517423 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.517386 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:32.707685 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.707651 5998 generic.go:356] "Generic (PLEG): container finished" podID="80f35655-46ac-4a54-8326-077b1c659773" containerID="5e6474eeded05740c4a7d52e4f315322151df794102e2172ab21c524016dad65" exitCode=0 Oct 30 02:15:32.707861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.707721 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" Oct 30 02:15:32.707861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.707767 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" event={"ID":"80f35655-46ac-4a54-8326-077b1c659773","Type":"ContainerDied","Data":"5e6474eeded05740c4a7d52e4f315322151df794102e2172ab21c524016dad65"} Oct 30 02:15:32.707861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.707801 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" event={"ID":"80f35655-46ac-4a54-8326-077b1c659773","Type":"ContainerDied","Data":"29e0408950de6ed3faa436d29fb39f0564358c30b5a98fbc1c0b88392b288425"} Oct 30 02:15:32.707861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.707823 5998 scope.go:122] "RemoveContainer" containerID="5e6474eeded05740c4a7d52e4f315322151df794102e2172ab21c524016dad65" Oct 30 02:15:32.721725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.721687 5998 scope.go:122] "RemoveContainer" containerID="5e6474eeded05740c4a7d52e4f315322151df794102e2172ab21c524016dad65" Oct 30 02:15:32.722065 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:32.722040 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e6474eeded05740c4a7d52e4f315322151df794102e2172ab21c524016dad65\": container with ID starting with 5e6474eeded05740c4a7d52e4f315322151df794102e2172ab21c524016dad65 not found: ID does not exist" containerID="5e6474eeded05740c4a7d52e4f315322151df794102e2172ab21c524016dad65" Oct 30 02:15:32.722191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.722072 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e6474eeded05740c4a7d52e4f315322151df794102e2172ab21c524016dad65"} err="failed to get container status \"5e6474eeded05740c4a7d52e4f315322151df794102e2172ab21c524016dad65\": rpc error: code = NotFound desc = could not find container \"5e6474eeded05740c4a7d52e4f315322151df794102e2172ab21c524016dad65\": container with ID starting with 5e6474eeded05740c4a7d52e4f315322151df794102e2172ab21c524016dad65 not found: ID does not exist" Oct 30 02:15:32.757700 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.757616 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq"] Oct 30 02:15:32.761119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.761075 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq"] Oct 30 02:15:32.828203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:32.828165 5998 scope.go:122] "RemoveContainer" containerID="8b47917fcd557c7c85013456921e4c1dcbce116d5eda88bc3a8957378978428a" Oct 30 02:15:33.089730 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:33.089627 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:33.133899 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:33.133850 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-584ec5/auth-test-model-584ec5-kserve-8589c66d4f-psgsq" podUID="80f35655-46ac-4a54-8326-077b1c659773" containerName="main" probeResult="failure" output="Get \"https://10.133.0.165:8000/ready\": context deadline exceeded" Oct 30 02:15:33.200033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:33.199995 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:34.419727 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:34.419669 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:34.556923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:34.556884 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f35655-46ac-4a54-8326-077b1c659773" path="/var/lib/kubelet/pods/80f35655-46ac-4a54-8326-077b1c659773/volumes" Oct 30 02:15:34.562868 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:34.562833 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:35.512403 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:35.512360 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:35.621074 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:35.621041 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:35.795167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:35.793800 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:15:35.795167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:35.793884 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" podUID="eef42836-3659-4fdc-b2ff-b8003e3e47a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:15:36.251407 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:36.251368 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:37.573180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:37.573144 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:37.782513 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:37.782480 5998 generic.go:356] "Generic (PLEG): container finished" podID="eef42836-3659-4fdc-b2ff-b8003e3e47a3" containerID="36eb90a21df0e911eec233e8ffa2718e87689656261903f0b350f12f26b420b4" exitCode=0 Oct 30 02:15:37.782707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:37.782556 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" event={"ID":"eef42836-3659-4fdc-b2ff-b8003e3e47a3","Type":"ContainerDied","Data":"36eb90a21df0e911eec233e8ffa2718e87689656261903f0b350f12f26b420b4"} Oct 30 02:15:37.783060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:37.783039 5998 scope.go:122] "RemoveContainer" containerID="36eb90a21df0e911eec233e8ffa2718e87689656261903f0b350f12f26b420b4" Oct 30 02:15:37.784403 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:37.784380 5998 generic.go:356] "Generic (PLEG): container finished" podID="8532131f-abe1-426d-9a42-36ad42fb7cdb" containerID="3cd7d398c5dc61ee87456fa1d7adbb155d01ac3f9c9d421d2f637ac3c163a3c5" exitCode=0 Oct 30 02:15:37.784516 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:37.784446 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" event={"ID":"8532131f-abe1-426d-9a42-36ad42fb7cdb","Type":"ContainerDied","Data":"3cd7d398c5dc61ee87456fa1d7adbb155d01ac3f9c9d421d2f637ac3c163a3c5"} Oct 30 02:15:37.784799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:37.784784 5998 scope.go:122] "RemoveContainer" containerID="3cd7d398c5dc61ee87456fa1d7adbb155d01ac3f9c9d421d2f637ac3c163a3c5" Oct 30 02:15:37.785033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:37.785006 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:15:37.785133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:37.785113 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:15:37.785202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:37.785171 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:15:37.786361 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:37.786334 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:15:37.786469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:37.786398 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:15:37.786469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:37.786438 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:15:37.841673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:37.841592 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:38.793446 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:38.793407 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" event={"ID":"eef42836-3659-4fdc-b2ff-b8003e3e47a3","Type":"ContainerStarted","Data":"4d81353fec7c40562c57163a6ac0d8fba1930d85d918a640fd769c1711949972"} Oct 30 02:15:38.795899 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:38.795873 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" event={"ID":"8532131f-abe1-426d-9a42-36ad42fb7cdb","Type":"ContainerStarted","Data":"2c760d1ac0802012f6dc21673b956e3af438973144bdc4d7def29f34f1257338"} Oct 30 02:15:38.851112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:38.850158 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:15:38.851112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:38.850349 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 30 02:15:38.851112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:38.850399 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" podUID="eef42836-3659-4fdc-b2ff-b8003e3e47a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 30 02:15:39.430120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:39.420008 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:39.545115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:39.544537 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:15:39.545115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:39.544668 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 02:15:39.545115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:39.544711 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" podUID="8532131f-abe1-426d-9a42-36ad42fb7cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 02:15:39.847219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:39.841561 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 30 02:15:39.847219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:39.841621 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" podUID="eef42836-3659-4fdc-b2ff-b8003e3e47a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 30 02:15:40.565174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:40.565127 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 02:15:40.565382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:40.565190 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" podUID="8532131f-abe1-426d-9a42-36ad42fb7cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 02:15:40.840171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:40.839429 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 30 02:15:40.840171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:40.839493 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" podUID="eef42836-3659-4fdc-b2ff-b8003e3e47a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 30 02:15:40.918116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:40.915523 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" podUID="5ec2625a-1e27-4543-aa0b-352230fbe1a8" containerName="main" containerID="cri-o://fe6062dda7cfea711a7816ecbd6c44d8c5e90d8ea0b9ec252273aca4aaac0bc9" gracePeriod=45 Oct 30 02:15:41.085734 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.085699 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:41.536116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.533475 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:15:41.565741 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.563228 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 02:15:41.565741 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.563292 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" podUID="8532131f-abe1-426d-9a42-36ad42fb7cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 02:15:41.685611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.685576 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-dshm\") pod \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " Oct 30 02:15:41.685807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.685644 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-home\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-home\") pod \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " Oct 30 02:15:41.686041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.685983 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-home" pod "5ec2625a-1e27-4543-aa0b-352230fbe1a8" (UID: "5ec2625a-1e27-4543-aa0b-352230fbe1a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:41.686153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.686115 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ec2625a-1e27-4543-aa0b-352230fbe1a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ec2625a-1e27-4543-aa0b-352230fbe1a8-tls-certs\") pod \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " Oct 30 02:15:41.686236 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.686213 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-model-cache\") pod \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " Oct 30 02:15:41.686303 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.686243 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec2625a-1e27-4543-aa0b-352230fbe1a8-kube-api-access-2pjdr\" (UniqueName: \"kubernetes.io/projected/5ec2625a-1e27-4543-aa0b-352230fbe1a8-kube-api-access-2pjdr\") pod \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " Oct 30 02:15:41.686491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.686466 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-model-cache" pod "5ec2625a-1e27-4543-aa0b-352230fbe1a8" (UID: "5ec2625a-1e27-4543-aa0b-352230fbe1a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:41.686758 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.686735 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-tmp-dir" pod "5ec2625a-1e27-4543-aa0b-352230fbe1a8" (UID: "5ec2625a-1e27-4543-aa0b-352230fbe1a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:41.686822 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.686770 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-tmp-dir\") pod \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\" (UID: \"5ec2625a-1e27-4543-aa0b-352230fbe1a8\") " Oct 30 02:15:41.687465 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.687444 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:41.687547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.687469 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:41.687547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.687499 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:41.688205 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.688167 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-dshm" pod "5ec2625a-1e27-4543-aa0b-352230fbe1a8" (UID: "5ec2625a-1e27-4543-aa0b-352230fbe1a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:41.688617 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.688593 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ec2625a-1e27-4543-aa0b-352230fbe1a8-tls-certs" pod "5ec2625a-1e27-4543-aa0b-352230fbe1a8" (UID: "5ec2625a-1e27-4543-aa0b-352230fbe1a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:15:41.689347 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.689317 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec2625a-1e27-4543-aa0b-352230fbe1a8-kube-api-access-2pjdr" pod "5ec2625a-1e27-4543-aa0b-352230fbe1a8" (UID: "5ec2625a-1e27-4543-aa0b-352230fbe1a8"). InnerVolumeSpecName "kube-api-access-2pjdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:41.789225 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.789187 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ec2625a-1e27-4543-aa0b-352230fbe1a8-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:41.789225 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.789217 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ec2625a-1e27-4543-aa0b-352230fbe1a8-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:41.789225 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.789230 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pjdr\" (UniqueName: \"kubernetes.io/projected/5ec2625a-1e27-4543-aa0b-352230fbe1a8-kube-api-access-2pjdr\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:15:41.847171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.845508 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 30 02:15:41.847171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.845599 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" podUID="eef42836-3659-4fdc-b2ff-b8003e3e47a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 30 02:15:41.873678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.873639 5998 generic.go:356] "Generic (PLEG): container finished" podID="5ec2625a-1e27-4543-aa0b-352230fbe1a8" containerID="fe6062dda7cfea711a7816ecbd6c44d8c5e90d8ea0b9ec252273aca4aaac0bc9" exitCode=0 Oct 30 02:15:41.873883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.873682 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" event={"ID":"5ec2625a-1e27-4543-aa0b-352230fbe1a8","Type":"ContainerDied","Data":"fe6062dda7cfea711a7816ecbd6c44d8c5e90d8ea0b9ec252273aca4aaac0bc9"} Oct 30 02:15:41.873883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.873734 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" event={"ID":"5ec2625a-1e27-4543-aa0b-352230fbe1a8","Type":"ContainerDied","Data":"075e919427cedbc27b41d7d70c7f73ab0cc2a0638eb2837f7ea09e1cb755e2d3"} Oct 30 02:15:41.873883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.873756 5998 scope.go:122] "RemoveContainer" containerID="fe6062dda7cfea711a7816ecbd6c44d8c5e90d8ea0b9ec252273aca4aaac0bc9" Oct 30 02:15:41.874186 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.874163 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" Oct 30 02:15:41.886370 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.886308 5998 scope.go:122] "RemoveContainer" containerID="fe6062dda7cfea711a7816ecbd6c44d8c5e90d8ea0b9ec252273aca4aaac0bc9" Oct 30 02:15:41.886676 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:41.886643 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe6062dda7cfea711a7816ecbd6c44d8c5e90d8ea0b9ec252273aca4aaac0bc9\": container with ID starting with fe6062dda7cfea711a7816ecbd6c44d8c5e90d8ea0b9ec252273aca4aaac0bc9 not found: ID does not exist" containerID="fe6062dda7cfea711a7816ecbd6c44d8c5e90d8ea0b9ec252273aca4aaac0bc9" Oct 30 02:15:41.886789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.886676 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe6062dda7cfea711a7816ecbd6c44d8c5e90d8ea0b9ec252273aca4aaac0bc9"} err="failed to get container status \"fe6062dda7cfea711a7816ecbd6c44d8c5e90d8ea0b9ec252273aca4aaac0bc9\": rpc error: code = NotFound desc = could not find container \"fe6062dda7cfea711a7816ecbd6c44d8c5e90d8ea0b9ec252273aca4aaac0bc9\": container with ID starting with fe6062dda7cfea711a7816ecbd6c44d8c5e90d8ea0b9ec252273aca4aaac0bc9 not found: ID does not exist" Oct 30 02:15:41.942200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.942163 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4"] Oct 30 02:15:41.974405 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:41.974233 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4"] Oct 30 02:15:42.449116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:42.448192 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-01a9cd/auth-test-model-01a9cd-kserve-d756499b5-rgxq4" podUID="5ec2625a-1e27-4543-aa0b-352230fbe1a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.166:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:15:42.543116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:42.542852 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 02:15:42.543116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:42.542916 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" podUID="8532131f-abe1-426d-9a42-36ad42fb7cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 02:15:42.564180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:42.564150 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec2625a-1e27-4543-aa0b-352230fbe1a8" path="/var/lib/kubelet/pods/5ec2625a-1e27-4543-aa0b-352230fbe1a8/volumes" Oct 30 02:15:42.678137 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:42.676996 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:42.841170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:42.840460 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 30 02:15:42.841170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:42.840522 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" podUID="eef42836-3659-4fdc-b2ff-b8003e3e47a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 30 02:15:43.537116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:43.534527 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 02:15:43.537116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:43.534590 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" podUID="8532131f-abe1-426d-9a42-36ad42fb7cdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 02:15:43.842180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:43.841621 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Oct 30 02:15:43.842180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:43.841718 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" podUID="eef42836-3659-4fdc-b2ff-b8003e3e47a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Oct 30 02:15:44.404416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:44.404374 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:44.519245 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:44.519207 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:44.591238 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:44.591186 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:15:44.594178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:44.591916 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:15:44.595365 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:44.594479 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:15:44.843971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:44.843933 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:15:44.844307 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:44.844284 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:15:44.845177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:44.845156 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:15:46.067286 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:46.067249 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:47.714894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:47.714859 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:49.333833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:49.333799 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:49.440994 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:49.440950 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:49.548140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:49.548083 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:51.002131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:51.000698 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:51.755959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:51.755921 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:52.611643 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:52.611605 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:52.911716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:52.911683 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:53.021615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:53.021584 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:53.206685 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:53.206590 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:53.514210 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:53.514124 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:53.672035 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:53.672001 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:53.911426 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:53.911369 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:54.292670 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:54.292547 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:54.410701 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:54.410660 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:54.520190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:54.520152 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:55.218921 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:15:55.218881 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:15:55.722550 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.722509 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:55.997139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997050 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7"] Oct 30 02:15:55.997659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997640 5998 cpu_manager.go:414] "containerMap: removing container" podUID="5ec2625a-1e27-4543-aa0b-352230fbe1a8" containerName="main" Oct 30 02:15:55.997746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997662 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec2625a-1e27-4543-aa0b-352230fbe1a8" containerName="main" Oct 30 02:15:55.997746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997687 5998 cpu_manager.go:414] "containerMap: removing container" podUID="30276c96-e1d6-4759-9368-51b819812537" containerName="cleanup" Oct 30 02:15:55.997746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997695 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30276c96-e1d6-4759-9368-51b819812537" containerName="cleanup" Oct 30 02:15:55.997746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997712 5998 cpu_manager.go:414] "containerMap: removing container" podUID="80f35655-46ac-4a54-8326-077b1c659773" containerName="main" Oct 30 02:15:55.997746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997720 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f35655-46ac-4a54-8326-077b1c659773" containerName="main" Oct 30 02:15:55.997746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997743 5998 cpu_manager.go:414] "containerMap: removing container" podUID="f70185f3-a978-4c11-ab66-903cfcba41b8" containerName="cleanup" Oct 30 02:15:55.998056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997750 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70185f3-a978-4c11-ab66-903cfcba41b8" containerName="cleanup" Oct 30 02:15:55.998056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997777 5998 cpu_manager.go:414] "containerMap: removing container" podUID="0a25d75a-85d3-4f3f-a951-aa39f35481fb" containerName="cleanup" Oct 30 02:15:55.998056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997784 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a25d75a-85d3-4f3f-a951-aa39f35481fb" containerName="cleanup" Oct 30 02:15:55.998056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997794 5998 cpu_manager.go:414] "containerMap: removing container" podUID="88d6d3ad-5fa9-4aca-9803-a981213c42af" containerName="cleanup" Oct 30 02:15:55.998056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997800 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d6d3ad-5fa9-4aca-9803-a981213c42af" containerName="cleanup" Oct 30 02:15:55.998056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997815 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3212debe-7561-4fa1-bdd9-bd34e88389ea" containerName="cleanup" Oct 30 02:15:55.998056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997821 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3212debe-7561-4fa1-bdd9-bd34e88389ea" containerName="cleanup" Oct 30 02:15:55.998056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997845 5998 cpu_manager.go:414] "containerMap: removing container" podUID="287b0a33-1dc7-4284-8c93-7af661cf9d37" containerName="cleanup" Oct 30 02:15:55.998056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997852 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287b0a33-1dc7-4284-8c93-7af661cf9d37" containerName="cleanup" Oct 30 02:15:55.998056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997861 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a93eb477-297b-4060-a79a-8ba36de67e83" containerName="cleanup" Oct 30 02:15:55.998056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997868 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93eb477-297b-4060-a79a-8ba36de67e83" containerName="cleanup" Oct 30 02:15:55.998056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997881 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ab563a5a-c1f8-4e32-aa5d-91effabc8a18" containerName="cleanup" Oct 30 02:15:55.998056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997888 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab563a5a-c1f8-4e32-aa5d-91effabc8a18" containerName="cleanup" Oct 30 02:15:55.998056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997900 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7a692aea-70f7-4f61-adc6-bba42c9cc1bd" containerName="cleanup" Oct 30 02:15:55.998056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.997906 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a692aea-70f7-4f61-adc6-bba42c9cc1bd" containerName="cleanup" Oct 30 02:15:55.998056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.998058 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a25d75a-85d3-4f3f-a951-aa39f35481fb" containerName="cleanup" Oct 30 02:15:55.999101 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.998070 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93eb477-297b-4060-a79a-8ba36de67e83" containerName="cleanup" Oct 30 02:15:55.999101 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.998082 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab563a5a-c1f8-4e32-aa5d-91effabc8a18" containerName="cleanup" Oct 30 02:15:55.999101 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.998111 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70185f3-a978-4c11-ab66-903cfcba41b8" containerName="cleanup" Oct 30 02:15:55.999101 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.998121 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e5a7ed-87b9-4be7-87fe-39944b774797" containerName="cleanup" Oct 30 02:15:55.999101 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.998133 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf51a9f7-5514-42fd-9122-75d1495cde92" containerName="cleanup" Oct 30 02:15:55.999101 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.998145 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea963033-f2cb-4e1c-a9ef-7d25c76581eb" containerName="cleanup" Oct 30 02:15:55.999101 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.998158 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec2625a-1e27-4543-aa0b-352230fbe1a8" containerName="main" Oct 30 02:15:55.999101 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:55.998170 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f35655-46ac-4a54-8326-077b1c659773" containerName="main" Oct 30 02:15:56.003636 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.003615 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" Oct 30 02:15:56.020888 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.020864 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0fdc4e\"/\"agc-cred-0952678f0032\"" Oct 30 02:15:56.035371 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.035342 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0fdc4e\"/\"kube-root-ca.crt\"" Oct 30 02:15:56.035574 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.035555 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0fdc4e\"/\"payload-processing-external-model-e2e--a5b60986-dockercfg-j22cz\"" Oct 30 02:15:56.035662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.035420 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0fdc4e\"/\"openshift-service-ca.crt\"" Oct 30 02:15:56.035662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.035438 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0fdc4e\"/\"routing-overlay\"" Oct 30 02:15:56.035662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.035346 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0fdc4e\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0fdc4e\"" Oct 30 02:15:56.041534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.041503 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7"] Oct 30 02:15:56.152406 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.152369 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91221a56-4509-40dc-aa54-66ebf1a0715c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7\" (UID: \"91221a56-4509-40dc-aa54-66ebf1a0715c\") " pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" Oct 30 02:15:56.154797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.154769 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91221a56-4509-40dc-aa54-66ebf1a0715c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7\" (UID: \"91221a56-4509-40dc-aa54-66ebf1a0715c\") " pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" Oct 30 02:15:56.158263 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.158232 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54l2z\" (UniqueName: \"kubernetes.io/projected/91221a56-4509-40dc-aa54-66ebf1a0715c-kube-api-access-54l2z\") pod \"payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7\" (UID: \"91221a56-4509-40dc-aa54-66ebf1a0715c\") " pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" Oct 30 02:15:56.158490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.158469 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91221a56-4509-40dc-aa54-66ebf1a0715c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7\" (UID: \"91221a56-4509-40dc-aa54-66ebf1a0715c\") " pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" Oct 30 02:15:56.182137 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.182082 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:56.263584 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.263498 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54l2z\" (UniqueName: \"kubernetes.io/projected/91221a56-4509-40dc-aa54-66ebf1a0715c-kube-api-access-54l2z\") pod \"payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7\" (UID: \"91221a56-4509-40dc-aa54-66ebf1a0715c\") " pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" Oct 30 02:15:56.286396 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.286351 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91221a56-4509-40dc-aa54-66ebf1a0715c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7\" (UID: \"91221a56-4509-40dc-aa54-66ebf1a0715c\") " pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" Oct 30 02:15:56.288207 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.287850 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91221a56-4509-40dc-aa54-66ebf1a0715c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7\" (UID: \"91221a56-4509-40dc-aa54-66ebf1a0715c\") " pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" Oct 30 02:15:56.288821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.288798 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91221a56-4509-40dc-aa54-66ebf1a0715c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7\" (UID: \"91221a56-4509-40dc-aa54-66ebf1a0715c\") " pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" Oct 30 02:15:56.290057 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.290034 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91221a56-4509-40dc-aa54-66ebf1a0715c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7\" (UID: \"91221a56-4509-40dc-aa54-66ebf1a0715c\") " pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" Oct 30 02:15:56.297282 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.297251 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91221a56-4509-40dc-aa54-66ebf1a0715c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7\" (UID: \"91221a56-4509-40dc-aa54-66ebf1a0715c\") " pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" Oct 30 02:15:56.307179 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.307150 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54l2z\" (UniqueName: \"kubernetes.io/projected/91221a56-4509-40dc-aa54-66ebf1a0715c-kube-api-access-54l2z\") pod \"payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7\" (UID: \"91221a56-4509-40dc-aa54-66ebf1a0715c\") " pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" Oct 30 02:15:56.316936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.316904 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91221a56-4509-40dc-aa54-66ebf1a0715c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7\" (UID: \"91221a56-4509-40dc-aa54-66ebf1a0715c\") " pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" Oct 30 02:15:56.322971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.322944 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" Oct 30 02:15:56.633004 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:56.632966 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7"] Oct 30 02:15:56.635882 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:15:56.635849 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91221a56_4509_40dc_aa54_66ebf1a0715c.slice/crio-9aa5f26e4dfa20562861d0fe1b869080b7341821110344f46c083568fd01dd6e WatchSource:0}: Error finding container 9aa5f26e4dfa20562861d0fe1b869080b7341821110344f46c083568fd01dd6e: Status 404 returned error can't find the container with id 9aa5f26e4dfa20562861d0fe1b869080b7341821110344f46c083568fd01dd6e Oct 30 02:15:57.032934 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:57.032891 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" event={"ID":"91221a56-4509-40dc-aa54-66ebf1a0715c","Type":"ContainerStarted","Data":"153aebfcb37a274f653e603ff39246afa9ec1146cce15fdf8a4037cc7200de8a"} Oct 30 02:15:57.033168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:57.032987 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" event={"ID":"91221a56-4509-40dc-aa54-66ebf1a0715c","Type":"ContainerStarted","Data":"9aa5f26e4dfa20562861d0fe1b869080b7341821110344f46c083568fd01dd6e"} Oct 30 02:15:57.033168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:57.033155 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" Oct 30 02:15:57.073626 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:57.073572 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" podStartSLOduration=2.073556671 podStartE2EDuration="2.073556671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:57.070629282 +0000 UTC m=+2786.649420363" watchObservedRunningTime="2026-10-30 02:15:57.073556671 +0000 UTC m=+2786.652347778" Oct 30 02:15:57.742580 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:57.742538 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:15:59.394320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:15:59.394281 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:00.975892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:00.975858 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:02.565680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:02.565644 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:03.149149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:03.149109 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:16:04.455855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:04.455735 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:06.240477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:06.240443 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:08.037553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:08.037511 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:08.044922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:08.044891 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" Oct 30 02:16:09.885159 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:09.885121 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:11.612542 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:11.612501 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:11.733160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:11.732075 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:11.941315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:11.941273 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:16:11.941491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:11.941388 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" podUID="d71cb5c7-54c2-42ed-9d33-de325c6c851a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:16:13.397770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:13.397729 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:15.022877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:15.022832 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:15.225403 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:15.225369 5998 generic.go:356] "Generic (PLEG): container finished" podID="d71cb5c7-54c2-42ed-9d33-de325c6c851a" containerID="ec6c449f8666fee5516ac536d6fc4ee05fe5053a80fe1061479126cf82b278b1" exitCode=0 Oct 30 02:16:15.225581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:15.225454 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" event={"ID":"d71cb5c7-54c2-42ed-9d33-de325c6c851a","Type":"ContainerDied","Data":"ec6c449f8666fee5516ac536d6fc4ee05fe5053a80fe1061479126cf82b278b1"} Oct 30 02:16:15.226015 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:15.225996 5998 scope.go:122] "RemoveContainer" containerID="ec6c449f8666fee5516ac536d6fc4ee05fe5053a80fe1061479126cf82b278b1" Oct 30 02:16:15.227965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:15.227933 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:16:15.228069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:15.228002 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:16:15.228069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:15.228035 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:16:16.240063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:16.240025 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" event={"ID":"d71cb5c7-54c2-42ed-9d33-de325c6c851a","Type":"ContainerStarted","Data":"11757bda7031d9f39d2f6e41e45495bee41e05f4bf0628b31bd9754ac1f6cbfb"} Oct 30 02:16:16.348554 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:16.348518 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:16:16.348768 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:16.348681 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 02:16:16.348768 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:16.348745 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" podUID="d71cb5c7-54c2-42ed-9d33-de325c6c851a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 02:16:16.596289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:16.596201 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:17.349044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:17.349007 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 02:16:17.349572 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:17.349067 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" podUID="d71cb5c7-54c2-42ed-9d33-de325c6c851a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 02:16:18.349113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:18.349057 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 02:16:18.349558 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:18.349128 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" podUID="d71cb5c7-54c2-42ed-9d33-de325c6c851a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 02:16:18.431233 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:18.430627 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:19.348375 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:19.348337 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 02:16:19.348569 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:19.348401 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" podUID="d71cb5c7-54c2-42ed-9d33-de325c6c851a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 02:16:20.210986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:20.210936 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:20.348819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:20.348774 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 02:16:20.349005 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:20.348840 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" podUID="d71cb5c7-54c2-42ed-9d33-de325c6c851a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 02:16:21.348625 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:21.348584 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 02:16:21.349199 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:21.348643 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" podUID="d71cb5c7-54c2-42ed-9d33-de325c6c851a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 02:16:21.869659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:21.869615 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:22.355483 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:22.355446 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:16:22.355968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:22.355944 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:16:22.356738 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:22.356714 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:16:23.385152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:23.385117 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:24.986500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:24.986468 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:25.218283 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:16:25.218245 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:16:26.507444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:26.507411 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:28.196295 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:28.196259 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:29.880608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:29.880572 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:30.248040 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:30.248008 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:31.375170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:31.375018 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:31.496808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:31.496769 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:31.608691 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:31.608653 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:31.721813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:31.721778 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:31.998489 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:31.998397 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:32.136149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:32.136109 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:32.247844 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:32.247804 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:32.742963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:32.742927 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:32.848481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:32.848448 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:33.555799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:33.555768 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:34.225573 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.225527 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:34.430162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.430084 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:34.487374 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.487300 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5"] Oct 30 02:16:34.492433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.492413 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" Oct 30 02:16:34.505756 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.505732 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"openshift-service-ca.crt\"" Oct 30 02:16:34.505980 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.505960 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"routing-overlay\"" Oct 30 02:16:34.506072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.505959 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"agc-cred-1991e0bfc294\"" Oct 30 02:16:34.516338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.516310 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"kube-root-ca.crt\"" Oct 30 02:16:34.516338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.516310 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-9dfb3c\"" Oct 30 02:16:34.516527 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.516343 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"payload-processing-external-model-e2e--1ed6558d-dockercfg-xn4z5\"" Oct 30 02:16:34.520801 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.520776 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5"] Oct 30 02:16:34.691112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.679980 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5\" (UID: \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" Oct 30 02:16:34.691112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.680168 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp5zn\" (UniqueName: \"kubernetes.io/projected/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-kube-api-access-pp5zn\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5\" (UID: \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" Oct 30 02:16:34.691112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.680223 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5\" (UID: \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" Oct 30 02:16:34.691112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.680312 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5\" (UID: \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" Oct 30 02:16:34.782190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.782051 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp5zn\" (UniqueName: \"kubernetes.io/projected/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-kube-api-access-pp5zn\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5\" (UID: \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" Oct 30 02:16:34.782190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.782144 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5\" (UID: \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" Oct 30 02:16:34.782465 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.782245 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5\" (UID: \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" Oct 30 02:16:34.782465 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.782364 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5\" (UID: \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" Oct 30 02:16:34.783079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.783051 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5\" (UID: \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" Oct 30 02:16:34.783237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.783215 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5\" (UID: \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" Oct 30 02:16:34.784810 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.784786 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5\" (UID: \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" Oct 30 02:16:34.803890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.803858 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp5zn\" (UniqueName: \"kubernetes.io/projected/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-kube-api-access-pp5zn\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5\" (UID: \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" Oct 30 02:16:34.807599 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:34.807571 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" Oct 30 02:16:35.138043 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:35.138017 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5"] Oct 30 02:16:35.140961 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:16:35.140924 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bbe6c0e_df36_46b9_ba3c_0c0f7cc83a1a.slice/crio-adf1afffad38ab05b76864f70a46e12940ac1baee0c26f93ddf6e74484b9241f WatchSource:0}: Error finding container adf1afffad38ab05b76864f70a46e12940ac1baee0c26f93ddf6e74484b9241f: Status 404 returned error can't find the container with id adf1afffad38ab05b76864f70a46e12940ac1baee0c26f93ddf6e74484b9241f Oct 30 02:16:35.345015 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:35.344980 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:35.437410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:35.437373 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" event={"ID":"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a","Type":"ContainerStarted","Data":"6b9352aa552701ce9d975eb440413e2104ddc60a34eef70e9f5f1bb1a6b25ab3"} Oct 30 02:16:35.437604 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:35.437435 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" event={"ID":"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a","Type":"ContainerStarted","Data":"adf1afffad38ab05b76864f70a46e12940ac1baee0c26f93ddf6e74484b9241f"} Oct 30 02:16:35.437604 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:35.437465 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" Oct 30 02:16:35.484153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:35.484041 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" podStartSLOduration=1.4840191200000001 podStartE2EDuration="1.48401912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:16:35.481280461 +0000 UTC m=+2825.060071535" watchObservedRunningTime="2026-10-30 02:16:35.48401912 +0000 UTC m=+2825.062810202" Oct 30 02:16:35.823852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:35.823760 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:36.273944 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.273890 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb"] Oct 30 02:16:36.278994 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.278958 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.292760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.292737 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-000edd-openshift-default-dockercfg-nq8tc\"" Oct 30 02:16:36.308672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.308633 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb"] Oct 30 02:16:36.319907 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.319866 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-data\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.320220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.320198 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-workload-certs\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.320416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.320399 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-credential-socket\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.320576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.320560 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istiod-ca-cert\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.320722 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.320708 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-workload-socket\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.320856 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.320842 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-envoy\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.321036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.321019 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-token\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.321235 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.321218 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l88f\" (UniqueName: \"kubernetes.io/projected/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-kube-api-access-2l88f\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.321394 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.321377 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-podinfo\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.344377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.344340 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:36.463582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.457314 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-credential-socket\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.463582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.457381 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istiod-ca-cert\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.463582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.457423 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-workload-socket\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.463582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.457448 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-envoy\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.463582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.457526 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-token\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.463582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.457578 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l88f\" (UniqueName: \"kubernetes.io/projected/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-kube-api-access-2l88f\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.463582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.457609 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-podinfo\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.463582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.457774 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-data\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.463582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.457861 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-workload-certs\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.463582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.458345 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-workload-certs\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.463582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.458576 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-credential-socket\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.463582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.459146 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istiod-ca-cert\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.463582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.459463 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-workload-socket\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.464807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.464357 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-data\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.529343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.519316 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-token\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.529343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.528161 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l88f\" (UniqueName: \"kubernetes.io/projected/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-kube-api-access-2l88f\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.530394 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.530367 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-envoy\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.558829 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.555658 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-podinfo\") pod \"e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.569144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.564910 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:36.596487 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.596170 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:36.947179 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.947146 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb"] Oct 30 02:16:36.950080 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:16:36.950048 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a29d403_91c3_4ab3_9f67_6d94fcfb557f.slice/crio-96c1404051f6058aa87b6a8d9d8c88346177822240f98a490804a45a22bb1397 WatchSource:0}: Error finding container 96c1404051f6058aa87b6a8d9d8c88346177822240f98a490804a45a22bb1397: Status 404 returned error can't find the container with id 96c1404051f6058aa87b6a8d9d8c88346177822240f98a490804a45a22bb1397 Oct 30 02:16:36.953516 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.953477 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:16:36.953647 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.953562 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:16:36.953647 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.953600 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:16:36.983614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:36.983574 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:37.516437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:37.516389 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" event={"ID":"5a29d403-91c3-4ab3-9f67-6d94fcfb557f","Type":"ContainerStarted","Data":"30897745dbb277e9949903b625671acd23a5b6555c79a8faecc52b2c6bc8bf30"} Oct 30 02:16:37.516912 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:37.516498 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" event={"ID":"5a29d403-91c3-4ab3-9f67-6d94fcfb557f","Type":"ContainerStarted","Data":"96c1404051f6058aa87b6a8d9d8c88346177822240f98a490804a45a22bb1397"} Oct 30 02:16:37.574945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:37.574889 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" podStartSLOduration=1.5748753130000002 podStartE2EDuration="1.574875313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:16:37.569816004 +0000 UTC m=+2827.148607085" watchObservedRunningTime="2026-10-30 02:16:37.574875313 +0000 UTC m=+2827.153666394" Oct 30 02:16:37.596901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:37.596862 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:37.705557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:37.705466 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:38.605271 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:38.605232 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:38.605757 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:38.605371 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:39.535638 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:39.535607 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:39.536630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:39.536606 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:16:39.948862 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:39.948816 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:40.059101 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:40.059068 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:40.294600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:40.294508 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:41.656301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:41.656263 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:41.777836 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:41.777792 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:42.025116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:42.022647 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:43.689742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:43.689707 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:45.266777 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:45.266742 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:46.507988 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:46.507959 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" Oct 30 02:16:46.757288 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:46.757253 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:16:46.980477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:46.980436 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:48.813067 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:48.813035 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:50.469989 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:50.469953 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:52.061440 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:52.061408 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:53.778346 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:53.778311 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:55.253104 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:16:55.253052 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:16:55.449611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:55.449573 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:57.047608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:57.047572 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:58.630497 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:58.630453 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:16:59.206971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:16:59.206931 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:00.209121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.209072 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:00.599471 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.599388 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg"] Oct 30 02:17:00.611403 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.611371 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.629293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.629265 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg"] Oct 30 02:17:00.629993 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.629964 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:00.631777 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.631750 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-9bf0e2\"/\"default-dockercfg-hss8w\"" Oct 30 02:17:00.631898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.631781 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-9bf0e2\"/\"explicit-model-9bf0e2-kserve-self-signed-certs\"" Oct 30 02:17:00.639687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.639657 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-9bf0e2\"/\"openshift-service-ca.crt\"" Oct 30 02:17:00.639809 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.639698 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-9bf0e2\"/\"kube-root-ca.crt\"" Oct 30 02:17:00.766868 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.766816 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-dshm\") pod \"explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.767067 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.766883 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-home\") pod \"explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.767171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.767139 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-model-cache\") pod \"explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.767309 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.767282 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkth5\" (UniqueName: \"kubernetes.io/projected/2d1ac843-f64d-48e5-b9b6-153fa22603ba-kube-api-access-xkth5\") pod \"explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.767422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.767405 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-tmp-dir\") pod \"explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.767464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.767436 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1ac843-f64d-48e5-b9b6-153fa22603ba-tls-certs\") pod \"explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.872619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.868945 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkth5\" (UniqueName: \"kubernetes.io/projected/2d1ac843-f64d-48e5-b9b6-153fa22603ba-kube-api-access-xkth5\") pod \"explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.872619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.869069 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-tmp-dir\") pod \"explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.872619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.869116 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1ac843-f64d-48e5-b9b6-153fa22603ba-tls-certs\") pod \"explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.872619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.869202 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-dshm\") pod \"explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.872619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.869237 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-home\") pod \"explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.872619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.869363 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-model-cache\") pod \"explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.872619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.869872 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-model-cache\") pod \"explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.872619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.870480 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-tmp-dir\") pod \"explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.872619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.871435 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-home\") pod \"explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.889195 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.876675 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1ac843-f64d-48e5-b9b6-153fa22603ba-tls-certs\") pod \"explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.913112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.905307 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-dshm\") pod \"explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.935500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.932145 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkth5\" (UniqueName: \"kubernetes.io/projected/2d1ac843-f64d-48e5-b9b6-153fa22603ba-kube-api-access-xkth5\") pod \"explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:00.957112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:00.944478 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:01.285268 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:01.285226 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg"] Oct 30 02:17:01.286879 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:17:01.286850 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d1ac843_f64d_48e5_b9b6_153fa22603ba.slice/crio-de75750a1c650894b1f64463d6d11d09f9f902d2ea8d953e0121ca4d3179e1f4 WatchSource:0}: Error finding container de75750a1c650894b1f64463d6d11d09f9f902d2ea8d953e0121ca4d3179e1f4: Status 404 returned error can't find the container with id de75750a1c650894b1f64463d6d11d09f9f902d2ea8d953e0121ca4d3179e1f4 Oct 30 02:17:01.729288 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:01.729243 5998 patch_prober.go:29] interesting pod/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:17:01.729502 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:01.729328 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" podUID="3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:17:01.868958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:01.868918 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:01.951450 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:01.951415 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" event={"ID":"2d1ac843-f64d-48e5-b9b6-153fa22603ba","Type":"ContainerStarted","Data":"d1d1e9769be72ab81b1d85b9d0d81b5774b64d681897cd97d30d53e51a1af78b"} Oct 30 02:17:01.951639 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:01.951465 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" event={"ID":"2d1ac843-f64d-48e5-b9b6-153fa22603ba","Type":"ContainerStarted","Data":"de75750a1c650894b1f64463d6d11d09f9f902d2ea8d953e0121ca4d3179e1f4"} Oct 30 02:17:01.991071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:01.990975 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" podStartSLOduration=1.990960411 podStartE2EDuration="1.990960411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:17:01.989447207 +0000 UTC m=+2851.568238287" watchObservedRunningTime="2026-10-30 02:17:01.990960411 +0000 UTC m=+2851.569751488" Oct 30 02:17:02.945065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:02.945026 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:03.568143 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:03.568023 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:03.969775 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:03.969734 5998 generic.go:356] "Generic (PLEG): container finished" podID="3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" containerID="1f94efb447b54eabe22a074dbc9fcb4c79542aa954598ae691ae82c5fcade00d" exitCode=0 Oct 30 02:17:03.970241 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:03.969802 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" event={"ID":"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff","Type":"ContainerDied","Data":"1f94efb447b54eabe22a074dbc9fcb4c79542aa954598ae691ae82c5fcade00d"} Oct 30 02:17:03.970365 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:03.970341 5998 scope.go:122] "RemoveContainer" containerID="1f94efb447b54eabe22a074dbc9fcb4c79542aa954598ae691ae82c5fcade00d" Oct 30 02:17:03.972282 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:03.972254 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:17:03.972379 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:03.972325 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:17:03.972379 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:03.972358 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:17:04.978851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:04.978816 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" event={"ID":"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff","Type":"ContainerStarted","Data":"e36ae957bab8284389ea80c79938cf916713eeacbc4126dd5958ee3af7f76503"} Oct 30 02:17:05.121638 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:05.121607 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:05.942502 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:05.942454 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:17:05.942699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:05.942663 5998 patch_prober.go:29] interesting pod/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 02:17:05.942755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:05.942725 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" podUID="3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 02:17:06.697417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:06.697386 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:06.942287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:06.942245 5998 patch_prober.go:29] interesting pod/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 02:17:06.942504 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:06.942327 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" podUID="3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 02:17:07.942492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:07.942434 5998 patch_prober.go:29] interesting pod/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 02:17:07.942902 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:07.942516 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" podUID="3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 02:17:08.283488 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:08.283408 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:08.942768 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:08.942732 5998 patch_prober.go:29] interesting pod/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 02:17:08.943331 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:08.943303 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" podUID="3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 02:17:09.746384 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:09.746340 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:09.942478 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:09.942427 5998 patch_prober.go:29] interesting pod/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 02:17:09.942692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:09.942489 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" podUID="3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 02:17:10.945278 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:10.945239 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:10.949212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:10.949183 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:17:10.949606 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:10.949584 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:17:10.950338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:10.950318 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:17:10.958889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:10.958867 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:11.069680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:11.069654 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:11.214444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:11.214333 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:12.687483 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:12.687444 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:13.423432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:13.423388 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:13.531178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:13.531118 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:14.261027 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:14.260987 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:15.867020 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:15.866962 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:17.446520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:17.446481 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:19.178698 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:19.178655 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:19.297919 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:19.297885 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:19.410651 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:19.410615 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:20.809781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:20.809747 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:21.585216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:21.585122 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg"] Oct 30 02:17:21.620080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:21.620024 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:22.088737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:22.088702 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:22.551578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:22.551513 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:22.674148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:22.674116 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:22.783670 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:22.783634 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:23.466190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:23.466158 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:23.580004 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:23.579961 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:24.118695 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.118664 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g"] Oct 30 02:17:24.124566 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.124543 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g" Oct 30 02:17:24.141553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.141518 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g"] Oct 30 02:17:24.147924 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.147893 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:17:24.148084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.147943 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:17:24.148084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.147998 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:17:24.148084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.148008 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:17:24.188891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.188857 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wfrd\" (UniqueName: \"kubernetes.io/projected/fb8ba250-629d-4170-8521-c4909c336dfb-kube-api-access-9wfrd\") pod \"maas-api-revoke-sub-27be6abd-sbf6g\" (UID: \"fb8ba250-629d-4170-8521-c4909c336dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g" Oct 30 02:17:24.189173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.188992 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8ba250-629d-4170-8521-c4909c336dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27be6abd-sbf6g\" (UID: \"fb8ba250-629d-4170-8521-c4909c336dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g" Oct 30 02:17:24.189256 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.189224 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8ba250-629d-4170-8521-c4909c336dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27be6abd-sbf6g\" (UID: \"fb8ba250-629d-4170-8521-c4909c336dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g" Oct 30 02:17:24.270625 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.270594 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:24.291065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.291029 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8ba250-629d-4170-8521-c4909c336dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27be6abd-sbf6g\" (UID: \"fb8ba250-629d-4170-8521-c4909c336dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g" Oct 30 02:17:24.291231 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.291198 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wfrd\" (UniqueName: \"kubernetes.io/projected/fb8ba250-629d-4170-8521-c4909c336dfb-kube-api-access-9wfrd\") pod \"maas-api-revoke-sub-27be6abd-sbf6g\" (UID: \"fb8ba250-629d-4170-8521-c4909c336dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g" Oct 30 02:17:24.291279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.291267 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8ba250-629d-4170-8521-c4909c336dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27be6abd-sbf6g\" (UID: \"fb8ba250-629d-4170-8521-c4909c336dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g" Oct 30 02:17:24.291860 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.291841 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8ba250-629d-4170-8521-c4909c336dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27be6abd-sbf6g\" (UID: \"fb8ba250-629d-4170-8521-c4909c336dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g" Oct 30 02:17:24.293661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.293643 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8ba250-629d-4170-8521-c4909c336dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27be6abd-sbf6g\" (UID: \"fb8ba250-629d-4170-8521-c4909c336dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g" Oct 30 02:17:24.325117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.324819 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wfrd\" (UniqueName: \"kubernetes.io/projected/fb8ba250-629d-4170-8521-c4909c336dfb-kube-api-access-9wfrd\") pod \"maas-api-revoke-sub-27be6abd-sbf6g\" (UID: \"fb8ba250-629d-4170-8521-c4909c336dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g" Oct 30 02:17:24.440759 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.440720 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g" Oct 30 02:17:24.745858 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:24.745706 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g"] Oct 30 02:17:24.748620 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:17:24.748561 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb8ba250_629d_4170_8521_c4909c336dfb.slice/crio-a489d9515b838e50b3331507736aae210b29b733c32b74e20c3f87f3916e53b0 WatchSource:0}: Error finding container a489d9515b838e50b3331507736aae210b29b733c32b74e20c3f87f3916e53b0: Status 404 returned error can't find the container with id a489d9515b838e50b3331507736aae210b29b733c32b74e20c3f87f3916e53b0 Oct 30 02:17:25.202633 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:25.202600 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g" event={"ID":"fb8ba250-629d-4170-8521-c4909c336dfb","Type":"ContainerStarted","Data":"b6525ea88038fc379ec0c73f962ce07280e2b4779fc8bd8b1fd209dbf390fd7c"} Oct 30 02:17:25.202851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:25.202660 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g" event={"ID":"fb8ba250-629d-4170-8521-c4909c336dfb","Type":"ContainerStarted","Data":"a489d9515b838e50b3331507736aae210b29b733c32b74e20c3f87f3916e53b0"} Oct 30 02:17:25.224176 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:17:25.224137 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:17:25.244235 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:25.244182 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g" podStartSLOduration=1.244162449 podStartE2EDuration="1.244162449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:17:25.239208087 +0000 UTC m=+2874.817999169" watchObservedRunningTime="2026-10-30 02:17:25.244162449 +0000 UTC m=+2874.822953529" Oct 30 02:17:25.884860 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:25.884826 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:26.212476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:26.212431 5998 generic.go:356] "Generic (PLEG): container finished" podID="fb8ba250-629d-4170-8521-c4909c336dfb" containerID="b6525ea88038fc379ec0c73f962ce07280e2b4779fc8bd8b1fd209dbf390fd7c" exitCode=0 Oct 30 02:17:26.212686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:26.212482 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g" event={"ID":"fb8ba250-629d-4170-8521-c4909c336dfb","Type":"ContainerDied","Data":"b6525ea88038fc379ec0c73f962ce07280e2b4779fc8bd8b1fd209dbf390fd7c"} Oct 30 02:17:27.032816 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:27.032774 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq"] Oct 30 02:17:27.036290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:27.036263 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4d1932a-dsnwq"] Oct 30 02:17:27.157226 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:27.157189 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:27.409441 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:27.409414 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g" Oct 30 02:17:27.475189 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:27.475153 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:27.543247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:27.539594 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb8ba250-629d-4170-8521-c4909c336dfb-kube-api-access-9wfrd\" (UniqueName: \"kubernetes.io/projected/fb8ba250-629d-4170-8521-c4909c336dfb-kube-api-access-9wfrd\") pod \"fb8ba250-629d-4170-8521-c4909c336dfb\" (UID: \"fb8ba250-629d-4170-8521-c4909c336dfb\") " Oct 30 02:17:27.554634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:27.549330 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb8ba250-629d-4170-8521-c4909c336dfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8ba250-629d-4170-8521-c4909c336dfb-maas-api-xks-ca\") pod \"fb8ba250-629d-4170-8521-c4909c336dfb\" (UID: \"fb8ba250-629d-4170-8521-c4909c336dfb\") " Oct 30 02:17:27.554634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:27.549514 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb8ba250-629d-4170-8521-c4909c336dfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8ba250-629d-4170-8521-c4909c336dfb-maas-api-openshift-ca\") pod \"fb8ba250-629d-4170-8521-c4909c336dfb\" (UID: \"fb8ba250-629d-4170-8521-c4909c336dfb\") " Oct 30 02:17:27.554634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:27.549963 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8ba250-629d-4170-8521-c4909c336dfb-kube-api-access-9wfrd" pod "fb8ba250-629d-4170-8521-c4909c336dfb" (UID: "fb8ba250-629d-4170-8521-c4909c336dfb"). InnerVolumeSpecName "kube-api-access-9wfrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:27.554634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:27.550332 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8ba250-629d-4170-8521-c4909c336dfb-maas-api-openshift-ca" pod "fb8ba250-629d-4170-8521-c4909c336dfb" (UID: "fb8ba250-629d-4170-8521-c4909c336dfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:27.554634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:27.550595 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb8ba250-629d-4170-8521-c4909c336dfb-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:17:27.554634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:27.550613 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wfrd\" (UniqueName: \"kubernetes.io/projected/fb8ba250-629d-4170-8521-c4909c336dfb-kube-api-access-9wfrd\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:17:27.555882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:27.555852 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8ba250-629d-4170-8521-c4909c336dfb-maas-api-xks-ca" pod "fb8ba250-629d-4170-8521-c4909c336dfb" (UID: "fb8ba250-629d-4170-8521-c4909c336dfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:27.652382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:27.652349 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb8ba250-629d-4170-8521-c4909c336dfb-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:17:28.234292 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:28.234252 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g" Oct 30 02:17:28.234824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:28.234290 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g" event={"ID":"fb8ba250-629d-4170-8521-c4909c336dfb","Type":"ContainerDied","Data":"a489d9515b838e50b3331507736aae210b29b733c32b74e20c3f87f3916e53b0"} Oct 30 02:17:28.234824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:28.234386 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a489d9515b838e50b3331507736aae210b29b733c32b74e20c3f87f3916e53b0" Oct 30 02:17:28.541013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:28.540930 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40108afb-04c9-46e4-9f0f-ef20676eb718" path="/var/lib/kubelet/pods/40108afb-04c9-46e4-9f0f-ef20676eb718/volumes" Oct 30 02:17:28.993553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:28.993518 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:30.491619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:30.491583 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:32.065608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:32.065570 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:33.628117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:33.628049 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:35.225902 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:35.225829 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:35.364333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:35.364294 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:35.471290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:35.471257 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:35.996030 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:35.995998 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:36.608985 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:36.608941 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" podUID="2d1ac843-f64d-48e5-b9b6-153fa22603ba" containerName="main" containerID="cri-o://d1d1e9769be72ab81b1d85b9d0d81b5774b64d681897cd97d30d53e51a1af78b" gracePeriod=45 Oct 30 02:17:36.910217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:36.910189 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:36.966055 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:36.966017 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:37.048883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.048847 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv"] Oct 30 02:17:37.053459 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.053417 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd779be9-5fggv"] Oct 30 02:17:37.092886 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.092850 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-model-cache\") pod \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " Oct 30 02:17:37.093064 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.092931 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-home\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-home\") pod \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " Oct 30 02:17:37.093064 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.092977 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1ac843-f64d-48e5-b9b6-153fa22603ba-kube-api-access-xkth5\" (UniqueName: \"kubernetes.io/projected/2d1ac843-f64d-48e5-b9b6-153fa22603ba-kube-api-access-xkth5\") pod \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " Oct 30 02:17:37.093223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.093201 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-home" pod "2d1ac843-f64d-48e5-b9b6-153fa22603ba" (UID: "2d1ac843-f64d-48e5-b9b6-153fa22603ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:37.093278 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.093217 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-model-cache" pod "2d1ac843-f64d-48e5-b9b6-153fa22603ba" (UID: "2d1ac843-f64d-48e5-b9b6-153fa22603ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:37.093278 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.093232 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-dshm\") pod \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " Oct 30 02:17:37.093361 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.093277 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d1ac843-f64d-48e5-b9b6-153fa22603ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1ac843-f64d-48e5-b9b6-153fa22603ba-tls-certs\") pod \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " Oct 30 02:17:37.093361 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.093325 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-tmp-dir\") pod \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\" (UID: \"2d1ac843-f64d-48e5-b9b6-153fa22603ba\") " Oct 30 02:17:37.093649 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.093625 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-tmp-dir" pod "2d1ac843-f64d-48e5-b9b6-153fa22603ba" (UID: "2d1ac843-f64d-48e5-b9b6-153fa22603ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:37.093989 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.093969 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:17:37.094080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.093993 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:17:37.094080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.094006 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:17:37.095854 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.095824 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d1ac843-f64d-48e5-b9b6-153fa22603ba-tls-certs" pod "2d1ac843-f64d-48e5-b9b6-153fa22603ba" (UID: "2d1ac843-f64d-48e5-b9b6-153fa22603ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:37.095854 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.095829 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-dshm" pod "2d1ac843-f64d-48e5-b9b6-153fa22603ba" (UID: "2d1ac843-f64d-48e5-b9b6-153fa22603ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:37.096031 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.095876 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1ac843-f64d-48e5-b9b6-153fa22603ba-kube-api-access-xkth5" pod "2d1ac843-f64d-48e5-b9b6-153fa22603ba" (UID: "2d1ac843-f64d-48e5-b9b6-153fa22603ba"). InnerVolumeSpecName "kube-api-access-xkth5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:37.196119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.196044 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1ac843-f64d-48e5-b9b6-153fa22603ba-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:17:37.196119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.196078 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1ac843-f64d-48e5-b9b6-153fa22603ba-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:17:37.196119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.196108 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkth5\" (UniqueName: \"kubernetes.io/projected/2d1ac843-f64d-48e5-b9b6-153fa22603ba-kube-api-access-xkth5\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:17:37.309219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.309184 5998 generic.go:356] "Generic (PLEG): container finished" podID="2d1ac843-f64d-48e5-b9b6-153fa22603ba" containerID="d1d1e9769be72ab81b1d85b9d0d81b5774b64d681897cd97d30d53e51a1af78b" exitCode=0 Oct 30 02:17:37.309415 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.309227 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" event={"ID":"2d1ac843-f64d-48e5-b9b6-153fa22603ba","Type":"ContainerDied","Data":"d1d1e9769be72ab81b1d85b9d0d81b5774b64d681897cd97d30d53e51a1af78b"} Oct 30 02:17:37.309415 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.309251 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" Oct 30 02:17:37.309415 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.309265 5998 scope.go:122] "RemoveContainer" containerID="d1d1e9769be72ab81b1d85b9d0d81b5774b64d681897cd97d30d53e51a1af78b" Oct 30 02:17:37.309415 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.309255 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg" event={"ID":"2d1ac843-f64d-48e5-b9b6-153fa22603ba","Type":"ContainerDied","Data":"de75750a1c650894b1f64463d6d11d09f9f902d2ea8d953e0121ca4d3179e1f4"} Oct 30 02:17:37.336045 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.336013 5998 scope.go:122] "RemoveContainer" containerID="d1d1e9769be72ab81b1d85b9d0d81b5774b64d681897cd97d30d53e51a1af78b" Oct 30 02:17:37.336471 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:17:37.336444 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1d1e9769be72ab81b1d85b9d0d81b5774b64d681897cd97d30d53e51a1af78b\": container with ID starting with d1d1e9769be72ab81b1d85b9d0d81b5774b64d681897cd97d30d53e51a1af78b not found: ID does not exist" containerID="d1d1e9769be72ab81b1d85b9d0d81b5774b64d681897cd97d30d53e51a1af78b" Oct 30 02:17:37.336572 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.336478 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1d1e9769be72ab81b1d85b9d0d81b5774b64d681897cd97d30d53e51a1af78b"} err="failed to get container status \"d1d1e9769be72ab81b1d85b9d0d81b5774b64d681897cd97d30d53e51a1af78b\": rpc error: code = NotFound desc = could not find container \"d1d1e9769be72ab81b1d85b9d0d81b5774b64d681897cd97d30d53e51a1af78b\": container with ID starting with d1d1e9769be72ab81b1d85b9d0d81b5774b64d681897cd97d30d53e51a1af78b not found: ID does not exist" Oct 30 02:17:37.363826 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.363794 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg"] Oct 30 02:17:37.369544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.369510 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-9bf0e2/explicit-model-9bf0e2-kserve-774b89d4c5-g6bgg"] Oct 30 02:17:37.834667 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:37.834631 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:38.539879 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:38.539842 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101c9f6c-b9b6-4408-873d-b0d14d3529b5" path="/var/lib/kubelet/pods/101c9f6c-b9b6-4408-873d-b0d14d3529b5/volumes" Oct 30 02:17:38.540335 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:38.540320 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d1ac843-f64d-48e5-b9b6-153fa22603ba" path="/var/lib/kubelet/pods/2d1ac843-f64d-48e5-b9b6-153fa22603ba/volumes" Oct 30 02:17:38.558634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:38.558595 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:39.792125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:39.792069 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:17:40.080425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:40.080292 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:41.912390 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:41.912104 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:43.272720 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:43.272675 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:43.438114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:43.432849 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:43.542730 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:43.542575 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:43.656102 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:43.656050 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:44.131190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.131154 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh"] Oct 30 02:17:44.131873 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.131847 5998 cpu_manager.go:414] "containerMap: removing container" podUID="fb8ba250-629d-4170-8521-c4909c336dfb" containerName="revoke-keys" Oct 30 02:17:44.131873 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.131872 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb8ba250-629d-4170-8521-c4909c336dfb" containerName="revoke-keys" Oct 30 02:17:44.132059 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.131922 5998 cpu_manager.go:414] "containerMap: removing container" podUID="2d1ac843-f64d-48e5-b9b6-153fa22603ba" containerName="main" Oct 30 02:17:44.132059 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.131931 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1ac843-f64d-48e5-b9b6-153fa22603ba" containerName="main" Oct 30 02:17:44.132221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.132130 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1ac843-f64d-48e5-b9b6-153fa22603ba" containerName="main" Oct 30 02:17:44.132221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.132158 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb8ba250-629d-4170-8521-c4909c336dfb" containerName="revoke-keys" Oct 30 02:17:44.137409 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.137387 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh" Oct 30 02:17:44.153250 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.153226 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh"] Oct 30 02:17:44.158179 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.158154 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:17:44.158344 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.158333 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:17:44.171396 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.170325 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:17:44.171396 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.170401 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:17:44.171396 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.171148 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcwbt\" (UniqueName: \"kubernetes.io/projected/6b9afa50-aa53-483b-96fd-327c6f0055ab-kube-api-access-vcwbt\") pod \"maas-api-revoke-sub-498006ff-xjwnh\" (UID: \"6b9afa50-aa53-483b-96fd-327c6f0055ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh" Oct 30 02:17:44.171396 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.171272 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9afa50-aa53-483b-96fd-327c6f0055ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-498006ff-xjwnh\" (UID: \"6b9afa50-aa53-483b-96fd-327c6f0055ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh" Oct 30 02:17:44.171396 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.171316 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9afa50-aa53-483b-96fd-327c6f0055ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-498006ff-xjwnh\" (UID: \"6b9afa50-aa53-483b-96fd-327c6f0055ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh" Oct 30 02:17:44.272433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.272395 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9afa50-aa53-483b-96fd-327c6f0055ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-498006ff-xjwnh\" (UID: \"6b9afa50-aa53-483b-96fd-327c6f0055ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh" Oct 30 02:17:44.272631 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.272443 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9afa50-aa53-483b-96fd-327c6f0055ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-498006ff-xjwnh\" (UID: \"6b9afa50-aa53-483b-96fd-327c6f0055ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh" Oct 30 02:17:44.272674 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.272641 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcwbt\" (UniqueName: \"kubernetes.io/projected/6b9afa50-aa53-483b-96fd-327c6f0055ab-kube-api-access-vcwbt\") pod \"maas-api-revoke-sub-498006ff-xjwnh\" (UID: \"6b9afa50-aa53-483b-96fd-327c6f0055ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh" Oct 30 02:17:44.273059 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.273023 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9afa50-aa53-483b-96fd-327c6f0055ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-498006ff-xjwnh\" (UID: \"6b9afa50-aa53-483b-96fd-327c6f0055ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh" Oct 30 02:17:44.275131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.275071 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9afa50-aa53-483b-96fd-327c6f0055ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-498006ff-xjwnh\" (UID: \"6b9afa50-aa53-483b-96fd-327c6f0055ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh" Oct 30 02:17:44.294525 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.294497 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcwbt\" (UniqueName: \"kubernetes.io/projected/6b9afa50-aa53-483b-96fd-327c6f0055ab-kube-api-access-vcwbt\") pod \"maas-api-revoke-sub-498006ff-xjwnh\" (UID: \"6b9afa50-aa53-483b-96fd-327c6f0055ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh" Oct 30 02:17:44.453534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.453495 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh" Oct 30 02:17:44.595835 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.595755 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:44.730188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.725771 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:44.894110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:44.886351 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh"] Oct 30 02:17:44.912281 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:17:44.912225 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9afa50_aa53_483b_96fd_327c6f0055ab.slice/crio-09a16901a30b074514a661befcc7e6c1293fe39593693b7d0d9a625b6bb672a3 WatchSource:0}: Error finding container 09a16901a30b074514a661befcc7e6c1293fe39593693b7d0d9a625b6bb672a3: Status 404 returned error can't find the container with id 09a16901a30b074514a661befcc7e6c1293fe39593693b7d0d9a625b6bb672a3 Oct 30 02:17:45.467381 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:45.467337 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh" event={"ID":"6b9afa50-aa53-483b-96fd-327c6f0055ab","Type":"ContainerStarted","Data":"09a16901a30b074514a661befcc7e6c1293fe39593693b7d0d9a625b6bb672a3"} Oct 30 02:17:45.522029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:45.521990 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:46.045332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.045296 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd"] Oct 30 02:17:46.050505 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.050481 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" Oct 30 02:17:46.064900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.064875 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"agc-cred-437db4025c2d\"" Oct 30 02:17:46.079276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.079245 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd"] Oct 30 02:17:46.163903 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.163861 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/539d48ea-7279-4f36-8f62-96587e06ef5d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd\" (UID: \"539d48ea-7279-4f36-8f62-96587e06ef5d\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" Oct 30 02:17:46.164110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.164009 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539d48ea-7279-4f36-8f62-96587e06ef5d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd\" (UID: \"539d48ea-7279-4f36-8f62-96587e06ef5d\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" Oct 30 02:17:46.164110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.164036 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/539d48ea-7279-4f36-8f62-96587e06ef5d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd\" (UID: \"539d48ea-7279-4f36-8f62-96587e06ef5d\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" Oct 30 02:17:46.164205 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.164141 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh4js\" (UniqueName: \"kubernetes.io/projected/539d48ea-7279-4f36-8f62-96587e06ef5d-kube-api-access-xh4js\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd\" (UID: \"539d48ea-7279-4f36-8f62-96587e06ef5d\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" Oct 30 02:17:46.265698 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.265660 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/539d48ea-7279-4f36-8f62-96587e06ef5d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd\" (UID: \"539d48ea-7279-4f36-8f62-96587e06ef5d\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" Oct 30 02:17:46.265883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.265859 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539d48ea-7279-4f36-8f62-96587e06ef5d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd\" (UID: \"539d48ea-7279-4f36-8f62-96587e06ef5d\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" Oct 30 02:17:46.265974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.265904 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/539d48ea-7279-4f36-8f62-96587e06ef5d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd\" (UID: \"539d48ea-7279-4f36-8f62-96587e06ef5d\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" Oct 30 02:17:46.266678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.266030 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh4js\" (UniqueName: \"kubernetes.io/projected/539d48ea-7279-4f36-8f62-96587e06ef5d-kube-api-access-xh4js\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd\" (UID: \"539d48ea-7279-4f36-8f62-96587e06ef5d\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" Oct 30 02:17:46.267130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.267039 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/539d48ea-7279-4f36-8f62-96587e06ef5d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd\" (UID: \"539d48ea-7279-4f36-8f62-96587e06ef5d\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" Oct 30 02:17:46.267212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.267142 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539d48ea-7279-4f36-8f62-96587e06ef5d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd\" (UID: \"539d48ea-7279-4f36-8f62-96587e06ef5d\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" Oct 30 02:17:46.269150 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.269130 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/539d48ea-7279-4f36-8f62-96587e06ef5d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd\" (UID: \"539d48ea-7279-4f36-8f62-96587e06ef5d\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" Oct 30 02:17:46.289505 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.289479 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh4js\" (UniqueName: \"kubernetes.io/projected/539d48ea-7279-4f36-8f62-96587e06ef5d-kube-api-access-xh4js\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd\" (UID: \"539d48ea-7279-4f36-8f62-96587e06ef5d\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" Oct 30 02:17:46.366149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.366114 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" Oct 30 02:17:46.483240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.483201 5998 generic.go:356] "Generic (PLEG): container finished" podID="6b9afa50-aa53-483b-96fd-327c6f0055ab" containerID="2742f2a02b2d5b241a089af1d2dd9a52598c53dafa4017c13086f7075bdd2bd4" exitCode=0 Oct 30 02:17:46.483705 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.483431 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh" event={"ID":"6b9afa50-aa53-483b-96fd-327c6f0055ab","Type":"ContainerDied","Data":"2742f2a02b2d5b241a089af1d2dd9a52598c53dafa4017c13086f7075bdd2bd4"} Oct 30 02:17:46.739111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:46.739033 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd"] Oct 30 02:17:46.744182 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:17:46.744151 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539d48ea_7279_4f36_8f62_96587e06ef5d.slice/crio-c4040ae087ac1c5e15bfcdedade1d7b0de6469010a9d23cad887f0fb4c4b4666 WatchSource:0}: Error finding container c4040ae087ac1c5e15bfcdedade1d7b0de6469010a9d23cad887f0fb4c4b4666: Status 404 returned error can't find the container with id c4040ae087ac1c5e15bfcdedade1d7b0de6469010a9d23cad887f0fb4c4b4666 Oct 30 02:17:47.229259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:47.224924 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:47.494364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:47.494272 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" event={"ID":"539d48ea-7279-4f36-8f62-96587e06ef5d","Type":"ContainerStarted","Data":"e88bdffb0366eac552613444563e16fee6bbc0cedabe056f13ad57bd712d932f"} Oct 30 02:17:47.495061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:47.495042 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" Oct 30 02:17:47.495166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:47.495152 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" event={"ID":"539d48ea-7279-4f36-8f62-96587e06ef5d","Type":"ContainerStarted","Data":"c4040ae087ac1c5e15bfcdedade1d7b0de6469010a9d23cad887f0fb4c4b4666"} Oct 30 02:17:47.541123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:47.540733 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" podStartSLOduration=1.540714506 podStartE2EDuration="1.540714506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:17:47.537075358 +0000 UTC m=+2897.115866439" watchObservedRunningTime="2026-10-30 02:17:47.540714506 +0000 UTC m=+2897.119505588" Oct 30 02:17:47.712352 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:47.712319 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh" Oct 30 02:17:47.789857 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:47.789764 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b9afa50-aa53-483b-96fd-327c6f0055ab-kube-api-access-vcwbt\" (UniqueName: \"kubernetes.io/projected/6b9afa50-aa53-483b-96fd-327c6f0055ab-kube-api-access-vcwbt\") pod \"6b9afa50-aa53-483b-96fd-327c6f0055ab\" (UID: \"6b9afa50-aa53-483b-96fd-327c6f0055ab\") " Oct 30 02:17:47.790038 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:47.789879 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b9afa50-aa53-483b-96fd-327c6f0055ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9afa50-aa53-483b-96fd-327c6f0055ab-maas-api-xks-ca\") pod \"6b9afa50-aa53-483b-96fd-327c6f0055ab\" (UID: \"6b9afa50-aa53-483b-96fd-327c6f0055ab\") " Oct 30 02:17:47.790038 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:47.790013 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b9afa50-aa53-483b-96fd-327c6f0055ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9afa50-aa53-483b-96fd-327c6f0055ab-maas-api-openshift-ca\") pod \"6b9afa50-aa53-483b-96fd-327c6f0055ab\" (UID: \"6b9afa50-aa53-483b-96fd-327c6f0055ab\") " Oct 30 02:17:47.791414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:47.791377 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b9afa50-aa53-483b-96fd-327c6f0055ab-maas-api-openshift-ca" pod "6b9afa50-aa53-483b-96fd-327c6f0055ab" (UID: "6b9afa50-aa53-483b-96fd-327c6f0055ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:47.798557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:47.798522 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9afa50-aa53-483b-96fd-327c6f0055ab-kube-api-access-vcwbt" pod "6b9afa50-aa53-483b-96fd-327c6f0055ab" (UID: "6b9afa50-aa53-483b-96fd-327c6f0055ab"). InnerVolumeSpecName "kube-api-access-vcwbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:47.803248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:47.803212 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b9afa50-aa53-483b-96fd-327c6f0055ab-maas-api-xks-ca" pod "6b9afa50-aa53-483b-96fd-327c6f0055ab" (UID: "6b9afa50-aa53-483b-96fd-327c6f0055ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:47.892201 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:47.892167 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcwbt\" (UniqueName: \"kubernetes.io/projected/6b9afa50-aa53-483b-96fd-327c6f0055ab-kube-api-access-vcwbt\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:17:47.892201 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:47.892199 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b9afa50-aa53-483b-96fd-327c6f0055ab-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:17:47.892201 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:47.892209 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b9afa50-aa53-483b-96fd-327c6f0055ab-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:17:48.505132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:48.503775 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh" event={"ID":"6b9afa50-aa53-483b-96fd-327c6f0055ab","Type":"ContainerDied","Data":"09a16901a30b074514a661befcc7e6c1293fe39593693b7d0d9a625b6bb672a3"} Oct 30 02:17:48.505132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:48.503800 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh" Oct 30 02:17:48.505132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:48.503841 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09a16901a30b074514a661befcc7e6c1293fe39593693b7d0d9a625b6bb672a3" Oct 30 02:17:48.785664 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:48.785587 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:49.728056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:49.728015 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:49.861643 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:49.861603 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:50.263821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:50.263753 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:50.380254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:50.380207 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:50.498013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:50.497974 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:51.376982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:51.376916 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:51.610804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:51.610701 5998 generic.go:356] "Generic (PLEG): container finished" podID="5a29d403-91c3-4ab3-9f67-6d94fcfb557f" containerID="30897745dbb277e9949903b625671acd23a5b6555c79a8faecc52b2c6bc8bf30" exitCode=0 Oct 30 02:17:51.610804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:51.610778 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" event={"ID":"5a29d403-91c3-4ab3-9f67-6d94fcfb557f","Type":"ContainerDied","Data":"30897745dbb277e9949903b625671acd23a5b6555c79a8faecc52b2c6bc8bf30"} Oct 30 02:17:51.611362 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:51.611340 5998 scope.go:122] "RemoveContainer" containerID="30897745dbb277e9949903b625671acd23a5b6555c79a8faecc52b2c6bc8bf30" Oct 30 02:17:51.613752 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:51.613713 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:17:51.613850 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:51.613806 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:17:51.613892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:51.613849 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:17:52.262413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:52.262378 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:52.621022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:52.620922 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" event={"ID":"5a29d403-91c3-4ab3-9f67-6d94fcfb557f","Type":"ContainerStarted","Data":"13b9b2e42b55b77b27259ea8d1a40b49d67703430e513b6907007b9afadf9d9f"} Oct 30 02:17:53.597223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:53.597189 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:17:53.598590 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:53.598559 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 02:17:53.598743 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:53.598620 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" podUID="5a29d403-91c3-4ab3-9f67-6d94fcfb557f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 02:17:53.971661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:53.971608 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:54.539063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:54.539034 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:17:54.597197 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:54.597162 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 02:17:54.597420 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:54.597224 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" podUID="5a29d403-91c3-4ab3-9f67-6d94fcfb557f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 02:17:55.033346 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:55.033307 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf"] Oct 30 02:17:55.043611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:55.043569 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f43081d9-g25sf"] Oct 30 02:17:55.249168 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:17:55.249125 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:17:55.458129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:55.451106 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:55.561665 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:55.561613 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:55.602553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:55.602511 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 02:17:55.602834 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:55.602803 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" podUID="5a29d403-91c3-4ab3-9f67-6d94fcfb557f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 02:17:55.791180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:55.788899 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:55.933947 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:55.933907 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7"] Oct 30 02:17:56.031881 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:56.031841 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:56.167542 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:56.167514 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:56.541492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:56.541406 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="677420bb-8553-4ae9-badd-fe881efddaf4" path="/var/lib/kubelet/pods/677420bb-8553-4ae9-badd-fe881efddaf4/volumes" Oct 30 02:17:56.597417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:56.597378 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 02:17:56.597635 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:56.597448 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" podUID="5a29d403-91c3-4ab3-9f67-6d94fcfb557f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 02:17:57.291693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:57.291662 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:57.597761 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:57.597678 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 02:17:57.597761 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:57.597731 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" podUID="5a29d403-91c3-4ab3-9f67-6d94fcfb557f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 02:17:58.602750 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:58.602718 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:17:58.603972 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:58.603947 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:17:58.879998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:58.879913 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:17:59.517608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:59.517574 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" Oct 30 02:17:59.598467 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:17:59.598432 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5"] Oct 30 02:18:00.490418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:00.490363 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:00.934549 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:00.934483 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" podUID="91221a56-4509-40dc-aa54-66ebf1a0715c" containerName="payload-processing" containerID="cri-o://153aebfcb37a274f653e603ff39246afa9ec1146cce15fdf8a4037cc7200de8a" gracePeriod=25 Oct 30 02:18:01.233780 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.233750 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" Oct 30 02:18:01.299448 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.299404 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91221a56-4509-40dc-aa54-66ebf1a0715c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91221a56-4509-40dc-aa54-66ebf1a0715c-routing-overlay-volume\") pod \"91221a56-4509-40dc-aa54-66ebf1a0715c\" (UID: \"91221a56-4509-40dc-aa54-66ebf1a0715c\") " Oct 30 02:18:01.299693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.299671 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91221a56-4509-40dc-aa54-66ebf1a0715c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/91221a56-4509-40dc-aa54-66ebf1a0715c-provider-credentials\") pod \"91221a56-4509-40dc-aa54-66ebf1a0715c\" (UID: \"91221a56-4509-40dc-aa54-66ebf1a0715c\") " Oct 30 02:18:01.299786 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.299765 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91221a56-4509-40dc-aa54-66ebf1a0715c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91221a56-4509-40dc-aa54-66ebf1a0715c-plugins-config-volume\") pod \"91221a56-4509-40dc-aa54-66ebf1a0715c\" (UID: \"91221a56-4509-40dc-aa54-66ebf1a0715c\") " Oct 30 02:18:01.299855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.299823 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91221a56-4509-40dc-aa54-66ebf1a0715c-kube-api-access-54l2z\" (UniqueName: \"kubernetes.io/projected/91221a56-4509-40dc-aa54-66ebf1a0715c-kube-api-access-54l2z\") pod \"91221a56-4509-40dc-aa54-66ebf1a0715c\" (UID: \"91221a56-4509-40dc-aa54-66ebf1a0715c\") " Oct 30 02:18:01.299933 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.299821 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91221a56-4509-40dc-aa54-66ebf1a0715c-routing-overlay-volume" pod "91221a56-4509-40dc-aa54-66ebf1a0715c" (UID: "91221a56-4509-40dc-aa54-66ebf1a0715c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:01.300486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.300239 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91221a56-4509-40dc-aa54-66ebf1a0715c-plugins-config-volume" pod "91221a56-4509-40dc-aa54-66ebf1a0715c" (UID: "91221a56-4509-40dc-aa54-66ebf1a0715c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:01.300486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.300437 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91221a56-4509-40dc-aa54-66ebf1a0715c-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:01.300486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.300458 5998 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91221a56-4509-40dc-aa54-66ebf1a0715c-routing-overlay-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:01.302405 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.302372 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91221a56-4509-40dc-aa54-66ebf1a0715c-provider-credentials" pod "91221a56-4509-40dc-aa54-66ebf1a0715c" (UID: "91221a56-4509-40dc-aa54-66ebf1a0715c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:01.302620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.302600 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91221a56-4509-40dc-aa54-66ebf1a0715c-kube-api-access-54l2z" pod "91221a56-4509-40dc-aa54-66ebf1a0715c" (UID: "91221a56-4509-40dc-aa54-66ebf1a0715c"). InnerVolumeSpecName "kube-api-access-54l2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:01.401937 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.401901 5998 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91221a56-4509-40dc-aa54-66ebf1a0715c-provider-credentials\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:01.401937 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.401932 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54l2z\" (UniqueName: \"kubernetes.io/projected/91221a56-4509-40dc-aa54-66ebf1a0715c-kube-api-access-54l2z\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:01.945290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.941709 5998 generic.go:356] "Generic (PLEG): container finished" podID="91221a56-4509-40dc-aa54-66ebf1a0715c" containerID="153aebfcb37a274f653e603ff39246afa9ec1146cce15fdf8a4037cc7200de8a" exitCode=0 Oct 30 02:18:01.945290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.941807 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" event={"ID":"91221a56-4509-40dc-aa54-66ebf1a0715c","Type":"ContainerDied","Data":"153aebfcb37a274f653e603ff39246afa9ec1146cce15fdf8a4037cc7200de8a"} Oct 30 02:18:01.945290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.941858 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" event={"ID":"91221a56-4509-40dc-aa54-66ebf1a0715c","Type":"ContainerDied","Data":"9aa5f26e4dfa20562861d0fe1b869080b7341821110344f46c083568fd01dd6e"} Oct 30 02:18:01.945290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.941886 5998 scope.go:122] "RemoveContainer" containerID="153aebfcb37a274f653e603ff39246afa9ec1146cce15fdf8a4037cc7200de8a" Oct 30 02:18:01.945290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.942066 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7" Oct 30 02:18:01.979326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.976229 5998 scope.go:122] "RemoveContainer" containerID="153aebfcb37a274f653e603ff39246afa9ec1146cce15fdf8a4037cc7200de8a" Oct 30 02:18:01.979326 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:18:01.976682 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153aebfcb37a274f653e603ff39246afa9ec1146cce15fdf8a4037cc7200de8a\": container with ID starting with 153aebfcb37a274f653e603ff39246afa9ec1146cce15fdf8a4037cc7200de8a not found: ID does not exist" containerID="153aebfcb37a274f653e603ff39246afa9ec1146cce15fdf8a4037cc7200de8a" Oct 30 02:18:01.979326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:01.976721 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153aebfcb37a274f653e603ff39246afa9ec1146cce15fdf8a4037cc7200de8a"} err="failed to get container status \"153aebfcb37a274f653e603ff39246afa9ec1146cce15fdf8a4037cc7200de8a\": rpc error: code = NotFound desc = could not find container \"153aebfcb37a274f653e603ff39246afa9ec1146cce15fdf8a4037cc7200de8a\": container with ID starting with 153aebfcb37a274f653e603ff39246afa9ec1146cce15fdf8a4037cc7200de8a not found: ID does not exist" Oct 30 02:18:02.008835 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:02.008798 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7"] Oct 30 02:18:02.012555 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:02.012521 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0fdc4e/payload-processing-external-model-e2e-extmod-b-0fdc4e-5c8dfdzv7"] Oct 30 02:18:02.201447 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:02.201354 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:02.542230 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:02.542149 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91221a56-4509-40dc-aa54-66ebf1a0715c" path="/var/lib/kubelet/pods/91221a56-4509-40dc-aa54-66ebf1a0715c/volumes" Oct 30 02:18:02.802022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:02.801934 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:02.913702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:02.913670 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:03.023030 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:03.022998 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:03.847116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:03.847072 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:04.553442 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.553397 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:04.599851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.599803 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" podUID="2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a" containerName="payload-processing" containerID="cri-o://6b9352aa552701ce9d975eb440413e2104ddc60a34eef70e9f5f1bb1a6b25ab3" gracePeriod=25 Oct 30 02:18:04.671069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.671031 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:04.878301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.878275 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" Oct 30 02:18:04.969804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.969767 5998 generic.go:356] "Generic (PLEG): container finished" podID="2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a" containerID="6b9352aa552701ce9d975eb440413e2104ddc60a34eef70e9f5f1bb1a6b25ab3" exitCode=0 Oct 30 02:18:04.969959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.969876 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" Oct 30 02:18:04.969959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.969906 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" event={"ID":"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a","Type":"ContainerDied","Data":"6b9352aa552701ce9d975eb440413e2104ddc60a34eef70e9f5f1bb1a6b25ab3"} Oct 30 02:18:04.970059 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.969966 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5" event={"ID":"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a","Type":"ContainerDied","Data":"adf1afffad38ab05b76864f70a46e12940ac1baee0c26f93ddf6e74484b9241f"} Oct 30 02:18:04.970059 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.969998 5998 scope.go:122] "RemoveContainer" containerID="6b9352aa552701ce9d975eb440413e2104ddc60a34eef70e9f5f1bb1a6b25ab3" Oct 30 02:18:04.977308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.977286 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-provider-credentials\") pod \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\" (UID: \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\") " Oct 30 02:18:04.977498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.977480 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-plugins-config-volume\") pod \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\" (UID: \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\") " Oct 30 02:18:04.977558 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.977540 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-routing-overlay-volume\") pod \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\" (UID: \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\") " Oct 30 02:18:04.977685 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.977663 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-kube-api-access-pp5zn\" (UniqueName: \"kubernetes.io/projected/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-kube-api-access-pp5zn\") pod \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\" (UID: \"2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a\") " Oct 30 02:18:04.977904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.977873 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-routing-overlay-volume" pod "2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a" (UID: "2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:04.978015 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.977930 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-plugins-config-volume" pod "2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a" (UID: "2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:04.978235 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.978205 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:04.978339 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.978237 5998 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-routing-overlay-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:04.979653 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.979628 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-provider-credentials" pod "2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a" (UID: "2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:04.979980 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.979955 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-kube-api-access-pp5zn" pod "2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a" (UID: "2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a"). InnerVolumeSpecName "kube-api-access-pp5zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:04.981004 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.980988 5998 scope.go:122] "RemoveContainer" containerID="6b9352aa552701ce9d975eb440413e2104ddc60a34eef70e9f5f1bb1a6b25ab3" Oct 30 02:18:04.981360 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:18:04.981339 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9352aa552701ce9d975eb440413e2104ddc60a34eef70e9f5f1bb1a6b25ab3\": container with ID starting with 6b9352aa552701ce9d975eb440413e2104ddc60a34eef70e9f5f1bb1a6b25ab3 not found: ID does not exist" containerID="6b9352aa552701ce9d975eb440413e2104ddc60a34eef70e9f5f1bb1a6b25ab3" Oct 30 02:18:04.981453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:04.981366 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9352aa552701ce9d975eb440413e2104ddc60a34eef70e9f5f1bb1a6b25ab3"} err="failed to get container status \"6b9352aa552701ce9d975eb440413e2104ddc60a34eef70e9f5f1bb1a6b25ab3\": rpc error: code = NotFound desc = could not find container \"6b9352aa552701ce9d975eb440413e2104ddc60a34eef70e9f5f1bb1a6b25ab3\": container with ID starting with 6b9352aa552701ce9d975eb440413e2104ddc60a34eef70e9f5f1bb1a6b25ab3 not found: ID does not exist" Oct 30 02:18:05.034696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:05.034659 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d"] Oct 30 02:18:05.039718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:05.039689 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44e724e-48n4d"] Oct 30 02:18:05.082111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:05.081256 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp5zn\" (UniqueName: \"kubernetes.io/projected/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-kube-api-access-pp5zn\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:05.104180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:05.081291 5998 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a-provider-credentials\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:05.315083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:05.315047 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5"] Oct 30 02:18:05.325293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:05.325258 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccxtxd5"] Oct 30 02:18:05.629379 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:05.629342 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:05.897218 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:05.897119 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:06.540100 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:06.540041 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a" path="/var/lib/kubelet/pods/2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a/volumes" Oct 30 02:18:06.540761 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:06.540732 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8" path="/var/lib/kubelet/pods/b12abfb1-c47a-4f71-bb25-ecfcf35bd7d8/volumes" Oct 30 02:18:06.912902 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:06.912862 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:07.277149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:07.277047 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:08.129483 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:08.129445 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:08.242315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:08.242283 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:08.411517 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:08.411484 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:08.707747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:08.707659 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:08.862864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:08.862823 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:09.035223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:09.035127 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:09.183146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:09.183078 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:09.510910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:09.510849 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:09.619553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:09.619494 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:10.789086 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:10.789029 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:10.897224 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:10.897189 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:11.015814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:11.015741 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:11.138342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:11.138240 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:11.323510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:11.323462 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:11.473342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:11.473310 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:12.935283 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:12.935199 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:13.072464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.071247 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:13.255011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.254838 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g"] Oct 30 02:18:13.259115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.259075 5998 cpu_manager.go:414] "containerMap: removing container" podUID="2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a" containerName="payload-processing" Oct 30 02:18:13.259323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.259305 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a" containerName="payload-processing" Oct 30 02:18:13.259438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.259426 5998 cpu_manager.go:414] "containerMap: removing container" podUID="6b9afa50-aa53-483b-96fd-327c6f0055ab" containerName="revoke-keys" Oct 30 02:18:13.259554 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.259541 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9afa50-aa53-483b-96fd-327c6f0055ab" containerName="revoke-keys" Oct 30 02:18:13.259683 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.259670 5998 cpu_manager.go:414] "containerMap: removing container" podUID="91221a56-4509-40dc-aa54-66ebf1a0715c" containerName="payload-processing" Oct 30 02:18:13.259799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.259786 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91221a56-4509-40dc-aa54-66ebf1a0715c" containerName="payload-processing" Oct 30 02:18:13.260735 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.260705 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bbe6c0e-df36-46b9-ba3c-0c0f7cc83a1a" containerName="payload-processing" Oct 30 02:18:13.260883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.260869 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9afa50-aa53-483b-96fd-327c6f0055ab" containerName="revoke-keys" Oct 30 02:18:13.260994 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.260982 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="91221a56-4509-40dc-aa54-66ebf1a0715c" containerName="payload-processing" Oct 30 02:18:13.266548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.266520 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" Oct 30 02:18:13.268098 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.268065 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g"] Oct 30 02:18:13.283839 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.283811 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"routing-overlay\"" Oct 30 02:18:13.283839 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.283836 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a0fc73\"" Oct 30 02:18:13.337028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.321270 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"openshift-service-ca.crt\"" Oct 30 02:18:13.337028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.321577 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"payload-processing-external-model-e2e--1bd5e061-dockercfg-txkmj\"" Oct 30 02:18:13.337028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.321706 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"e2e-cmp-legacy-a7974f-provider-cred\"" Oct 30 02:18:13.351121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.338287 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"kube-root-ca.crt\"" Oct 30 02:18:13.455112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.451302 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g\" (UID: \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" Oct 30 02:18:13.455112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.451373 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g\" (UID: \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" Oct 30 02:18:13.455112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.451528 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g\" (UID: \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" Oct 30 02:18:13.455112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.451854 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pn67\" (UniqueName: \"kubernetes.io/projected/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-kube-api-access-8pn67\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g\" (UID: \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" Oct 30 02:18:13.591172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.588082 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pn67\" (UniqueName: \"kubernetes.io/projected/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-kube-api-access-8pn67\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g\" (UID: \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" Oct 30 02:18:13.591172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.588267 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g\" (UID: \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" Oct 30 02:18:13.591172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.588304 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g\" (UID: \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" Oct 30 02:18:13.591172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.588375 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g\" (UID: \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" Oct 30 02:18:13.591172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.589779 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g\" (UID: \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" Oct 30 02:18:13.591172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.590724 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g\" (UID: \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" Oct 30 02:18:13.594534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.594506 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g\" (UID: \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" Oct 30 02:18:13.617057 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.614118 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pn67\" (UniqueName: \"kubernetes.io/projected/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-kube-api-access-8pn67\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g\" (UID: \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" Oct 30 02:18:13.778973 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.778940 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:13.884627 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:13.884542 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" Oct 30 02:18:14.196646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:14.196511 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g"] Oct 30 02:18:14.199546 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:18:14.199503 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8174b8d_1a7a_4ca2_800f_0f90f1a2e2a4.slice/crio-0cca2eaeb93f84c1694055b5921673b58a84235b5155f6381bb2dc9bc5ba0d6e WatchSource:0}: Error finding container 0cca2eaeb93f84c1694055b5921673b58a84235b5155f6381bb2dc9bc5ba0d6e: Status 404 returned error can't find the container with id 0cca2eaeb93f84c1694055b5921673b58a84235b5155f6381bb2dc9bc5ba0d6e Oct 30 02:18:14.480168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:14.475270 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:15.112264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:15.112218 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" event={"ID":"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4","Type":"ContainerStarted","Data":"471a41f1e3af2d2e018072cb11709b8f8fdf6a7983607e69ab0e7ba70e7ef177"} Oct 30 02:18:15.112600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:15.112558 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" event={"ID":"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4","Type":"ContainerStarted","Data":"0cca2eaeb93f84c1694055b5921673b58a84235b5155f6381bb2dc9bc5ba0d6e"} Oct 30 02:18:15.112718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:15.112612 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" Oct 30 02:18:15.154527 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:15.154463 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" podStartSLOduration=2.154448021 podStartE2EDuration="2.154448021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:18:15.15091282 +0000 UTC m=+2924.729703903" watchObservedRunningTime="2026-10-30 02:18:15.154448021 +0000 UTC m=+2924.733239103" Oct 30 02:18:15.163884 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:15.163853 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:16.034041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:16.034004 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l"] Oct 30 02:18:16.041450 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:16.041409 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-076fce0f-clx8l"] Oct 30 02:18:16.070300 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:16.070253 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:16.180024 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:16.179977 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:16.575114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:16.575059 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a405c38-2529-417f-94da-c49e43adb00b" path="/var/lib/kubelet/pods/3a405c38-2529-417f-94da-c49e43adb00b/volumes" Oct 30 02:18:17.062110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:17.059702 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:18.562970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:18.562928 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:18.814827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:18.814711 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb"] Oct 30 02:18:18.815080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:18.815053 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:18.819190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:18.819172 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" Oct 30 02:18:18.835085 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:18.835060 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"payload-processing-external-model-e2e--b7f31742-dockercfg-lxnpj\"" Oct 30 02:18:18.845788 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:18.845758 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6b8bfa\"" Oct 30 02:18:18.845928 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:18.845766 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"routing-overlay\"" Oct 30 02:18:18.845928 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:18.845766 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"kube-root-ca.crt\"" Oct 30 02:18:18.845928 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:18.845806 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"openshift-service-ca.crt\"" Oct 30 02:18:18.852267 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:18.852242 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb"] Oct 30 02:18:18.925580 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:18.925542 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f586b65e-70b2-422a-a095-26ffde3d6fe4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb\" (UID: \"f586b65e-70b2-422a-a095-26ffde3d6fe4\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" Oct 30 02:18:18.925779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:18.925600 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph9bd\" (UniqueName: \"kubernetes.io/projected/f586b65e-70b2-422a-a095-26ffde3d6fe4-kube-api-access-ph9bd\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb\" (UID: \"f586b65e-70b2-422a-a095-26ffde3d6fe4\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" Oct 30 02:18:18.925836 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:18.925807 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f586b65e-70b2-422a-a095-26ffde3d6fe4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb\" (UID: \"f586b65e-70b2-422a-a095-26ffde3d6fe4\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" Oct 30 02:18:18.925942 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:18.925925 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f586b65e-70b2-422a-a095-26ffde3d6fe4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb\" (UID: \"f586b65e-70b2-422a-a095-26ffde3d6fe4\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" Oct 30 02:18:18.935937 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:18.935914 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6b8bfa\"/\"agc-cred-721b1cf2e75f\"" Oct 30 02:18:19.029114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:19.028224 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f586b65e-70b2-422a-a095-26ffde3d6fe4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb\" (UID: \"f586b65e-70b2-422a-a095-26ffde3d6fe4\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" Oct 30 02:18:19.029114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:19.028365 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f586b65e-70b2-422a-a095-26ffde3d6fe4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb\" (UID: \"f586b65e-70b2-422a-a095-26ffde3d6fe4\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" Oct 30 02:18:19.029114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:19.028689 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f586b65e-70b2-422a-a095-26ffde3d6fe4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb\" (UID: \"f586b65e-70b2-422a-a095-26ffde3d6fe4\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" Oct 30 02:18:19.029418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:19.029398 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9bd\" (UniqueName: \"kubernetes.io/projected/f586b65e-70b2-422a-a095-26ffde3d6fe4-kube-api-access-ph9bd\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb\" (UID: \"f586b65e-70b2-422a-a095-26ffde3d6fe4\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" Oct 30 02:18:19.031079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:19.031039 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f586b65e-70b2-422a-a095-26ffde3d6fe4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb\" (UID: \"f586b65e-70b2-422a-a095-26ffde3d6fe4\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" Oct 30 02:18:19.032156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:19.032135 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f586b65e-70b2-422a-a095-26ffde3d6fe4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb\" (UID: \"f586b65e-70b2-422a-a095-26ffde3d6fe4\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" Oct 30 02:18:19.068012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:19.067927 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f586b65e-70b2-422a-a095-26ffde3d6fe4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb\" (UID: \"f586b65e-70b2-422a-a095-26ffde3d6fe4\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" Oct 30 02:18:19.088828 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:19.088785 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph9bd\" (UniqueName: \"kubernetes.io/projected/f586b65e-70b2-422a-a095-26ffde3d6fe4-kube-api-access-ph9bd\") pod \"payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb\" (UID: \"f586b65e-70b2-422a-a095-26ffde3d6fe4\") " pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" Oct 30 02:18:19.142788 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:19.142490 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" Oct 30 02:18:19.463390 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:19.463353 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb"] Oct 30 02:18:19.466391 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:18:19.466359 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf586b65e_70b2_422a_a095_26ffde3d6fe4.slice/crio-4883b17b99fc666ee4a4bd3a68bc281d9a90ab1644e6d895785881d04a930f8d WatchSource:0}: Error finding container 4883b17b99fc666ee4a4bd3a68bc281d9a90ab1644e6d895785881d04a930f8d: Status 404 returned error can't find the container with id 4883b17b99fc666ee4a4bd3a68bc281d9a90ab1644e6d895785881d04a930f8d Oct 30 02:18:20.176851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:20.176807 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" event={"ID":"f586b65e-70b2-422a-a095-26ffde3d6fe4","Type":"ContainerStarted","Data":"33959555a72a2acc615634c0d9ae7b658a441f0123070e4da2d599665f64a87e"} Oct 30 02:18:20.177349 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:20.176867 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" event={"ID":"f586b65e-70b2-422a-a095-26ffde3d6fe4","Type":"ContainerStarted","Data":"4883b17b99fc666ee4a4bd3a68bc281d9a90ab1644e6d895785881d04a930f8d"} Oct 30 02:18:20.177349 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:20.176936 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" Oct 30 02:18:20.217616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:20.217562 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" podStartSLOduration=2.217547224 podStartE2EDuration="2.217547224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:18:20.214523726 +0000 UTC m=+2929.793314824" watchObservedRunningTime="2026-10-30 02:18:20.217547224 +0000 UTC m=+2929.796338664" Oct 30 02:18:20.371574 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:20.371542 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:20.714060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:20.714016 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:20.821374 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:20.821338 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:21.355364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:21.355329 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:21.722705 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:21.722667 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2"] Oct 30 02:18:21.723252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:21.723217 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" podUID="3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" containerName="istio-proxy" containerID="cri-o://e36ae957bab8284389ea80c79938cf916713eeacbc4126dd5958ee3af7f76503" gracePeriod=30 Oct 30 02:18:21.774753 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:21.774716 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:22.270496 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:22.270441 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:23.684415 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:23.684372 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:23.891547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:23.891511 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:24.009262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.009184 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb"] Oct 30 02:18:24.012186 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.012159 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" Oct 30 02:18:24.096219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.096142 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:24.112876 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.111135 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7"] Oct 30 02:18:24.118409 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.118378 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.135875 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.135792 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-95f40f-openshift-default-dockercfg-85kw8\"" Oct 30 02:18:24.138171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.138142 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7"] Oct 30 02:18:24.208842 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.208806 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:24.231577 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.231538 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-envoy\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.231803 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.231592 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-credential-socket\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.231803 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.231696 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnbl2\" (UniqueName: \"kubernetes.io/projected/7076db80-f9a3-423f-b198-f0a7aba80bd7-kube-api-access-qnbl2\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.232287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.231823 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7076db80-f9a3-423f-b198-f0a7aba80bd7-istiod-ca-cert\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.232287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.232013 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-podinfo\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.232287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.232041 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-token\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.232287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.232260 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-data\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.232530 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.232321 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-workload-socket\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.232530 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.232386 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-workload-certs\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.333350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.333254 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-envoy\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.333350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.333291 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-credential-socket\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.333603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.333471 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnbl2\" (UniqueName: \"kubernetes.io/projected/7076db80-f9a3-423f-b198-f0a7aba80bd7-kube-api-access-qnbl2\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.333603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.333572 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7076db80-f9a3-423f-b198-f0a7aba80bd7-istiod-ca-cert\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.333760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.333732 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-credential-socket\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.334347 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.334316 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7076db80-f9a3-423f-b198-f0a7aba80bd7-istiod-ca-cert\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.334574 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.334552 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-podinfo\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.334638 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.334589 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-token\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.334703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.334666 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-data\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.335142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.334870 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-workload-socket\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.335142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.334911 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-workload-certs\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.335142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.334982 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-data\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.335377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.335228 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-workload-socket\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.335444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.335402 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-workload-certs\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.336015 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.335991 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-envoy\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.337730 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.337706 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-podinfo\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.356302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.356245 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-token\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.356494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.356470 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnbl2\" (UniqueName: \"kubernetes.io/projected/7076db80-f9a3-423f-b198-f0a7aba80bd7-kube-api-access-qnbl2\") pod \"e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.411000 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.410958 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:24.445147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.437285 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:24.981197 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.981167 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7"] Oct 30 02:18:24.983269 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:18:24.983241 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7076db80_f9a3_423f_b198_f0a7aba80bd7.slice/crio-61da4ffb1bb1590733c271d8a64fab8ba8ac9b855b2902a58e30dde4b3cc5000 WatchSource:0}: Error finding container 61da4ffb1bb1590733c271d8a64fab8ba8ac9b855b2902a58e30dde4b3cc5000: Status 404 returned error can't find the container with id 61da4ffb1bb1590733c271d8a64fab8ba8ac9b855b2902a58e30dde4b3cc5000 Oct 30 02:18:24.986938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.986896 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:18:24.987048 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.986998 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:18:24.987048 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:24.987042 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:18:25.222872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:25.222769 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" event={"ID":"7076db80-f9a3-423f-b198-f0a7aba80bd7","Type":"ContainerStarted","Data":"912eab5621d876448ae3aa2a0207002feac417c12e21aa8cae2226c48ff37021"} Oct 30 02:18:25.222872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:25.222839 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" event={"ID":"7076db80-f9a3-423f-b198-f0a7aba80bd7","Type":"ContainerStarted","Data":"61da4ffb1bb1590733c271d8a64fab8ba8ac9b855b2902a58e30dde4b3cc5000"} Oct 30 02:18:25.234197 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:18:25.229923 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:18:25.267443 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:25.267391 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" podStartSLOduration=1.267376995 podStartE2EDuration="1.267376995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:18:25.265923737 +0000 UTC m=+2934.844714818" watchObservedRunningTime="2026-10-30 02:18:25.267376995 +0000 UTC m=+2934.846168076" Oct 30 02:18:25.439116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:25.437506 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:25.628912 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:25.628805 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:25.952231 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:25.952192 5998 patch_prober.go:29] interesting pod/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:18:25.952413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:25.952271 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" podUID="3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:18:26.126650 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:26.126619 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" Oct 30 02:18:26.446039 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:26.446002 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:27.013434 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.013401 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:18:27.045243 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.045201 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb"] Oct 30 02:18:27.049996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.049963 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d5195dc-qrgmb"] Oct 30 02:18:27.078152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.078073 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-kube-api-access-b5l79\" (UniqueName: \"kubernetes.io/projected/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-kube-api-access-b5l79\") pod \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " Oct 30 02:18:27.078333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.078216 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-podinfo\") pod \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " Oct 30 02:18:27.078333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.078279 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-data\") pod \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " Oct 30 02:18:27.078469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.078330 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-workload-socket\") pod \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " Oct 30 02:18:27.078469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.078373 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-envoy\") pod \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " Oct 30 02:18:27.078611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.078562 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-data" pod "3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" (UID: "3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:27.078673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.078627 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-token\" (UniqueName: \"kubernetes.io/projected/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-token\") pod \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " Oct 30 02:18:27.078739 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.078721 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istiod-ca-cert\") pod \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " Oct 30 02:18:27.078792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.078775 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-credential-socket\") pod \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " Oct 30 02:18:27.078840 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.078823 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-workload-certs\") pod \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\" (UID: \"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff\") " Oct 30 02:18:27.079491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.079464 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:27.079728 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.079705 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-workload-certs" pod "3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" (UID: "3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:27.079994 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.079958 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-credential-socket" pod "3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" (UID: "3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:27.080111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.080025 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istiod-ca-cert" pod "3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" (UID: "3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:27.080111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.080039 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-workload-socket" pod "3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" (UID: "3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:27.080968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.080944 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-kube-api-access-b5l79" pod "3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" (UID: "3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff"). InnerVolumeSpecName "kube-api-access-b5l79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:27.081082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.081004 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-envoy" pod "3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" (UID: "3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:27.081959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.081927 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-podinfo" pod "3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" (UID: "3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:18:27.083436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.083409 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-token" pod "3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" (UID: "3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:27.181126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.181078 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:27.181126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.181121 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:27.181126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.181131 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:27.181652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.181141 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:27.181652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.181150 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5l79\" (UniqueName: \"kubernetes.io/projected/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-kube-api-access-b5l79\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:27.181652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.181159 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:27.181652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.181167 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:27.181652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.181176 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:27.245206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.245108 5998 generic.go:356] "Generic (PLEG): container finished" podID="3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" containerID="e36ae957bab8284389ea80c79938cf916713eeacbc4126dd5958ee3af7f76503" exitCode=0 Oct 30 02:18:27.245206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.245161 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" Oct 30 02:18:27.245206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.245188 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" event={"ID":"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff","Type":"ContainerDied","Data":"e36ae957bab8284389ea80c79938cf916713eeacbc4126dd5958ee3af7f76503"} Oct 30 02:18:27.245500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.245262 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2" event={"ID":"3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff","Type":"ContainerDied","Data":"0015764ffd8a1b70445ba61034f831838c17e1f170fb4f32f5c8ce1a6148b7b2"} Oct 30 02:18:27.245500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.245292 5998 scope.go:122] "RemoveContainer" containerID="e36ae957bab8284389ea80c79938cf916713eeacbc4126dd5958ee3af7f76503" Oct 30 02:18:27.246313 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.246290 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:27.247172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.247147 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:18:27.258319 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.258298 5998 scope.go:122] "RemoveContainer" containerID="1f94efb447b54eabe22a074dbc9fcb4c79542aa954598ae691ae82c5fcade00d" Oct 30 02:18:27.268650 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.268633 5998 scope.go:122] "RemoveContainer" containerID="e36ae957bab8284389ea80c79938cf916713eeacbc4126dd5958ee3af7f76503" Oct 30 02:18:27.268939 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:18:27.268921 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36ae957bab8284389ea80c79938cf916713eeacbc4126dd5958ee3af7f76503\": container with ID starting with e36ae957bab8284389ea80c79938cf916713eeacbc4126dd5958ee3af7f76503 not found: ID does not exist" containerID="e36ae957bab8284389ea80c79938cf916713eeacbc4126dd5958ee3af7f76503" Oct 30 02:18:27.269026 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.268947 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36ae957bab8284389ea80c79938cf916713eeacbc4126dd5958ee3af7f76503"} err="failed to get container status \"e36ae957bab8284389ea80c79938cf916713eeacbc4126dd5958ee3af7f76503\": rpc error: code = NotFound desc = could not find container \"e36ae957bab8284389ea80c79938cf916713eeacbc4126dd5958ee3af7f76503\": container with ID starting with e36ae957bab8284389ea80c79938cf916713eeacbc4126dd5958ee3af7f76503 not found: ID does not exist" Oct 30 02:18:27.269026 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.268963 5998 scope.go:122] "RemoveContainer" containerID="1f94efb447b54eabe22a074dbc9fcb4c79542aa954598ae691ae82c5fcade00d" Oct 30 02:18:27.269204 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:18:27.269186 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f94efb447b54eabe22a074dbc9fcb4c79542aa954598ae691ae82c5fcade00d\": container with ID starting with 1f94efb447b54eabe22a074dbc9fcb4c79542aa954598ae691ae82c5fcade00d not found: ID does not exist" containerID="1f94efb447b54eabe22a074dbc9fcb4c79542aa954598ae691ae82c5fcade00d" Oct 30 02:18:27.269265 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.269212 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f94efb447b54eabe22a074dbc9fcb4c79542aa954598ae691ae82c5fcade00d"} err="failed to get container status \"1f94efb447b54eabe22a074dbc9fcb4c79542aa954598ae691ae82c5fcade00d\": rpc error: code = NotFound desc = could not find container \"1f94efb447b54eabe22a074dbc9fcb4c79542aa954598ae691ae82c5fcade00d\": container with ID starting with 1f94efb447b54eabe22a074dbc9fcb4c79542aa954598ae691ae82c5fcade00d not found: ID does not exist" Oct 30 02:18:27.348156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.346811 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2"] Oct 30 02:18:27.352204 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.352168 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-9bf0e2-openshift-default-59c74b64b9-47xt2"] Oct 30 02:18:27.376472 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.376439 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:27.612027 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.611938 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:27.730002 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:27.729970 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:28.540670 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:28.540631 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" path="/var/lib/kubelet/pods/3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff/volumes" Oct 30 02:18:28.541376 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:28.541352 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c2b7588-719c-4b04-9a95-963da9214886" path="/var/lib/kubelet/pods/7c2b7588-719c-4b04-9a95-963da9214886/volumes" Oct 30 02:18:29.010134 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.010080 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" podUID="f586b65e-70b2-422a-a095-26ffde3d6fe4" containerName="payload-processing" containerID="cri-o://33959555a72a2acc615634c0d9ae7b658a441f0123070e4da2d599665f64a87e" gracePeriod=25 Oct 30 02:18:29.215656 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.215622 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:29.294426 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.294318 5998 generic.go:356] "Generic (PLEG): container finished" podID="f586b65e-70b2-422a-a095-26ffde3d6fe4" containerID="33959555a72a2acc615634c0d9ae7b658a441f0123070e4da2d599665f64a87e" exitCode=0 Oct 30 02:18:29.294600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.294422 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" event={"ID":"f586b65e-70b2-422a-a095-26ffde3d6fe4","Type":"ContainerDied","Data":"33959555a72a2acc615634c0d9ae7b658a441f0123070e4da2d599665f64a87e"} Oct 30 02:18:29.341110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.340450 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" Oct 30 02:18:29.341110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.341003 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:29.415056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.415022 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f586b65e-70b2-422a-a095-26ffde3d6fe4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f586b65e-70b2-422a-a095-26ffde3d6fe4-plugins-config-volume\") pod \"f586b65e-70b2-422a-a095-26ffde3d6fe4\" (UID: \"f586b65e-70b2-422a-a095-26ffde3d6fe4\") " Oct 30 02:18:29.415280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.415180 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f586b65e-70b2-422a-a095-26ffde3d6fe4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f586b65e-70b2-422a-a095-26ffde3d6fe4-provider-credentials\") pod \"f586b65e-70b2-422a-a095-26ffde3d6fe4\" (UID: \"f586b65e-70b2-422a-a095-26ffde3d6fe4\") " Oct 30 02:18:29.415280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.415264 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f586b65e-70b2-422a-a095-26ffde3d6fe4-kube-api-access-ph9bd\" (UniqueName: \"kubernetes.io/projected/f586b65e-70b2-422a-a095-26ffde3d6fe4-kube-api-access-ph9bd\") pod \"f586b65e-70b2-422a-a095-26ffde3d6fe4\" (UID: \"f586b65e-70b2-422a-a095-26ffde3d6fe4\") " Oct 30 02:18:29.415427 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.415370 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f586b65e-70b2-422a-a095-26ffde3d6fe4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f586b65e-70b2-422a-a095-26ffde3d6fe4-routing-overlay-volume\") pod \"f586b65e-70b2-422a-a095-26ffde3d6fe4\" (UID: \"f586b65e-70b2-422a-a095-26ffde3d6fe4\") " Oct 30 02:18:29.415589 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.415560 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f586b65e-70b2-422a-a095-26ffde3d6fe4-plugins-config-volume" pod "f586b65e-70b2-422a-a095-26ffde3d6fe4" (UID: "f586b65e-70b2-422a-a095-26ffde3d6fe4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:29.416070 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.416041 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f586b65e-70b2-422a-a095-26ffde3d6fe4-routing-overlay-volume" pod "f586b65e-70b2-422a-a095-26ffde3d6fe4" (UID: "f586b65e-70b2-422a-a095-26ffde3d6fe4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:29.416209 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.416077 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f586b65e-70b2-422a-a095-26ffde3d6fe4-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:29.418893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.418476 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f586b65e-70b2-422a-a095-26ffde3d6fe4-kube-api-access-ph9bd" pod "f586b65e-70b2-422a-a095-26ffde3d6fe4" (UID: "f586b65e-70b2-422a-a095-26ffde3d6fe4"). InnerVolumeSpecName "kube-api-access-ph9bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:29.418893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.418699 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f586b65e-70b2-422a-a095-26ffde3d6fe4-provider-credentials" pod "f586b65e-70b2-422a-a095-26ffde3d6fe4" (UID: "f586b65e-70b2-422a-a095-26ffde3d6fe4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:29.449059 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.449022 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:29.517183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.517151 5998 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f586b65e-70b2-422a-a095-26ffde3d6fe4-routing-overlay-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:29.517183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.517180 5998 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f586b65e-70b2-422a-a095-26ffde3d6fe4-provider-credentials\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:29.517183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.517189 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph9bd\" (UniqueName: \"kubernetes.io/projected/f586b65e-70b2-422a-a095-26ffde3d6fe4-kube-api-access-ph9bd\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:29.721114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:29.720601 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd"] Oct 30 02:18:30.303268 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:30.303222 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" Oct 30 02:18:30.303268 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:30.303248 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb" event={"ID":"f586b65e-70b2-422a-a095-26ffde3d6fe4","Type":"ContainerDied","Data":"4883b17b99fc666ee4a4bd3a68bc281d9a90ab1644e6d895785881d04a930f8d"} Oct 30 02:18:30.303557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:30.303323 5998 scope.go:122] "RemoveContainer" containerID="33959555a72a2acc615634c0d9ae7b658a441f0123070e4da2d599665f64a87e" Oct 30 02:18:30.355884 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:30.355849 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb"] Oct 30 02:18:30.366684 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:30.366654 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6b8bfa/payload-processing-external-model-e2e-extmod-a-6b8bfa-6cdbqd2cb"] Oct 30 02:18:30.541789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:30.541752 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f586b65e-70b2-422a-a095-26ffde3d6fe4" path="/var/lib/kubelet/pods/f586b65e-70b2-422a-a095-26ffde3d6fe4/volumes" Oct 30 02:18:30.939884 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:30.939845 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:32.558861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:32.558826 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:33.032070 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:33.032032 5998 scope.go:122] "RemoveContainer" containerID="f891d4a7ef591e5c691a64019ef5de17613ae4529a3d6440c9ea589d1531e33a" Oct 30 02:18:34.096971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:34.096934 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:34.721534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:34.721485 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" podUID="539d48ea-7279-4f36-8f62-96587e06ef5d" containerName="payload-processing" containerID="cri-o://e88bdffb0366eac552613444563e16fee6bbc0cedabe056f13ad57bd712d932f" gracePeriod=25 Oct 30 02:18:34.990972 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:34.990910 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" Oct 30 02:18:35.122749 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.122714 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/539d48ea-7279-4f36-8f62-96587e06ef5d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/539d48ea-7279-4f36-8f62-96587e06ef5d-provider-credentials\") pod \"539d48ea-7279-4f36-8f62-96587e06ef5d\" (UID: \"539d48ea-7279-4f36-8f62-96587e06ef5d\") " Oct 30 02:18:35.123306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.122948 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/539d48ea-7279-4f36-8f62-96587e06ef5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539d48ea-7279-4f36-8f62-96587e06ef5d-plugins-config-volume\") pod \"539d48ea-7279-4f36-8f62-96587e06ef5d\" (UID: \"539d48ea-7279-4f36-8f62-96587e06ef5d\") " Oct 30 02:18:35.123306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.123025 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/539d48ea-7279-4f36-8f62-96587e06ef5d-kube-api-access-xh4js\" (UniqueName: \"kubernetes.io/projected/539d48ea-7279-4f36-8f62-96587e06ef5d-kube-api-access-xh4js\") pod \"539d48ea-7279-4f36-8f62-96587e06ef5d\" (UID: \"539d48ea-7279-4f36-8f62-96587e06ef5d\") " Oct 30 02:18:35.123306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.123081 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/539d48ea-7279-4f36-8f62-96587e06ef5d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/539d48ea-7279-4f36-8f62-96587e06ef5d-routing-overlay-volume\") pod \"539d48ea-7279-4f36-8f62-96587e06ef5d\" (UID: \"539d48ea-7279-4f36-8f62-96587e06ef5d\") " Oct 30 02:18:35.123486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.123441 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/539d48ea-7279-4f36-8f62-96587e06ef5d-plugins-config-volume" pod "539d48ea-7279-4f36-8f62-96587e06ef5d" (UID: "539d48ea-7279-4f36-8f62-96587e06ef5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:35.123577 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.123548 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/539d48ea-7279-4f36-8f62-96587e06ef5d-routing-overlay-volume" pod "539d48ea-7279-4f36-8f62-96587e06ef5d" (UID: "539d48ea-7279-4f36-8f62-96587e06ef5d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:35.123774 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.123756 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/539d48ea-7279-4f36-8f62-96587e06ef5d-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:35.123840 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.123779 5998 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/539d48ea-7279-4f36-8f62-96587e06ef5d-routing-overlay-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:35.125133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.125074 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/539d48ea-7279-4f36-8f62-96587e06ef5d-provider-credentials" pod "539d48ea-7279-4f36-8f62-96587e06ef5d" (UID: "539d48ea-7279-4f36-8f62-96587e06ef5d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:35.125265 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.125141 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/539d48ea-7279-4f36-8f62-96587e06ef5d-kube-api-access-xh4js" pod "539d48ea-7279-4f36-8f62-96587e06ef5d" (UID: "539d48ea-7279-4f36-8f62-96587e06ef5d"). InnerVolumeSpecName "kube-api-access-xh4js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:35.225491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.225455 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh4js\" (UniqueName: \"kubernetes.io/projected/539d48ea-7279-4f36-8f62-96587e06ef5d-kube-api-access-xh4js\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:35.225491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.225489 5998 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/539d48ea-7279-4f36-8f62-96587e06ef5d-provider-credentials\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:18:35.350443 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.350356 5998 generic.go:356] "Generic (PLEG): container finished" podID="539d48ea-7279-4f36-8f62-96587e06ef5d" containerID="e88bdffb0366eac552613444563e16fee6bbc0cedabe056f13ad57bd712d932f" exitCode=0 Oct 30 02:18:35.350443 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.350410 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" Oct 30 02:18:35.350443 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.350411 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" event={"ID":"539d48ea-7279-4f36-8f62-96587e06ef5d","Type":"ContainerDied","Data":"e88bdffb0366eac552613444563e16fee6bbc0cedabe056f13ad57bd712d932f"} Oct 30 02:18:35.350721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.350486 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd" event={"ID":"539d48ea-7279-4f36-8f62-96587e06ef5d","Type":"ContainerDied","Data":"c4040ae087ac1c5e15bfcdedade1d7b0de6469010a9d23cad887f0fb4c4b4666"} Oct 30 02:18:35.350721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.350518 5998 scope.go:122] "RemoveContainer" containerID="e88bdffb0366eac552613444563e16fee6bbc0cedabe056f13ad57bd712d932f" Oct 30 02:18:35.363632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.363607 5998 scope.go:122] "RemoveContainer" containerID="e88bdffb0366eac552613444563e16fee6bbc0cedabe056f13ad57bd712d932f" Oct 30 02:18:35.363964 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:18:35.363940 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88bdffb0366eac552613444563e16fee6bbc0cedabe056f13ad57bd712d932f\": container with ID starting with e88bdffb0366eac552613444563e16fee6bbc0cedabe056f13ad57bd712d932f not found: ID does not exist" containerID="e88bdffb0366eac552613444563e16fee6bbc0cedabe056f13ad57bd712d932f" Oct 30 02:18:35.364062 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.363975 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88bdffb0366eac552613444563e16fee6bbc0cedabe056f13ad57bd712d932f"} err="failed to get container status \"e88bdffb0366eac552613444563e16fee6bbc0cedabe056f13ad57bd712d932f\": rpc error: code = NotFound desc = could not find container \"e88bdffb0366eac552613444563e16fee6bbc0cedabe056f13ad57bd712d932f\": container with ID starting with e88bdffb0366eac552613444563e16fee6bbc0cedabe056f13ad57bd712d932f not found: ID does not exist" Oct 30 02:18:35.397836 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.397778 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd"] Oct 30 02:18:35.400915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.400874 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd49g6gd"] Oct 30 02:18:35.690589 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.690553 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:35.818932 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:35.818890 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:18:36.539632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:36.539601 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="539d48ea-7279-4f36-8f62-96587e06ef5d" path="/var/lib/kubelet/pods/539d48ea-7279-4f36-8f62-96587e06ef5d/volumes" Oct 30 02:18:37.587454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:37.587417 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:37.971245 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:37.971207 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:39.261991 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:39.261958 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:40.702825 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:40.702787 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:40.955147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:40.955033 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:42.692042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:42.692008 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:44.452172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:44.452137 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:46.022171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:46.022132 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:47.689045 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:47.689003 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:49.471859 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:49.471824 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:51.046425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:51.046387 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:52.569675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:52.569640 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:54.191049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:54.191012 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:55.222577 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:18:55.222532 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:18:55.762730 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:55.762697 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:55.868033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:55.867991 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:57.342678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:57.342637 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:18:59.012238 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:18:59.012199 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:00.671288 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:00.671249 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:02.209702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:02.209661 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:03.696216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:03.696178 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:05.383111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:05.383043 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:07.023238 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:07.023204 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:07.156114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:07.153714 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:07.332082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:07.331998 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:08.776922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:08.776883 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:09.174266 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.174227 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4"] Oct 30 02:19:09.174829 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.174812 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" containerName="istio-proxy" Oct 30 02:19:09.174878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.174829 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" containerName="istio-proxy" Oct 30 02:19:09.174878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.174838 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" containerName="istio-proxy" Oct 30 02:19:09.174878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.174843 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" containerName="istio-proxy" Oct 30 02:19:09.174878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.174859 5998 cpu_manager.go:414] "containerMap: removing container" podUID="539d48ea-7279-4f36-8f62-96587e06ef5d" containerName="payload-processing" Oct 30 02:19:09.174878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.174863 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="539d48ea-7279-4f36-8f62-96587e06ef5d" containerName="payload-processing" Oct 30 02:19:09.175047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.174887 5998 cpu_manager.go:414] "containerMap: removing container" podUID="f586b65e-70b2-422a-a095-26ffde3d6fe4" containerName="payload-processing" Oct 30 02:19:09.175047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.174892 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f586b65e-70b2-422a-a095-26ffde3d6fe4" containerName="payload-processing" Oct 30 02:19:09.175047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.174969 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="f586b65e-70b2-422a-a095-26ffde3d6fe4" containerName="payload-processing" Oct 30 02:19:09.175047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.174981 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" containerName="istio-proxy" Oct 30 02:19:09.175047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.174988 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e65fd4e-e7a0-4124-88cf-c08e9f83a5ff" containerName="istio-proxy" Oct 30 02:19:09.175047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.174995 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="539d48ea-7279-4f36-8f62-96587e06ef5d" containerName="payload-processing" Oct 30 02:19:09.178510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.178492 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4" Oct 30 02:19:09.191283 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.191264 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:19:09.201766 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.201739 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:19:09.201766 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.201756 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:19:09.201932 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.201771 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:19:09.205813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.205789 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4"] Oct 30 02:19:09.244147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.244110 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446df54f-cdf8-4604-a08b-6b9b603f370f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-400b9185-dhzc4\" (UID: \"446df54f-cdf8-4604-a08b-6b9b603f370f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4" Oct 30 02:19:09.244588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.244546 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446df54f-cdf8-4604-a08b-6b9b603f370f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-400b9185-dhzc4\" (UID: \"446df54f-cdf8-4604-a08b-6b9b603f370f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4" Oct 30 02:19:09.244731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.244641 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhqsc\" (UniqueName: \"kubernetes.io/projected/446df54f-cdf8-4604-a08b-6b9b603f370f-kube-api-access-xhqsc\") pod \"maas-api-revoke-sub-400b9185-dhzc4\" (UID: \"446df54f-cdf8-4604-a08b-6b9b603f370f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4" Oct 30 02:19:09.345702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.345649 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446df54f-cdf8-4604-a08b-6b9b603f370f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-400b9185-dhzc4\" (UID: \"446df54f-cdf8-4604-a08b-6b9b603f370f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4" Oct 30 02:19:09.345980 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.345745 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhqsc\" (UniqueName: \"kubernetes.io/projected/446df54f-cdf8-4604-a08b-6b9b603f370f-kube-api-access-xhqsc\") pod \"maas-api-revoke-sub-400b9185-dhzc4\" (UID: \"446df54f-cdf8-4604-a08b-6b9b603f370f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4" Oct 30 02:19:09.345980 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.345944 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446df54f-cdf8-4604-a08b-6b9b603f370f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-400b9185-dhzc4\" (UID: \"446df54f-cdf8-4604-a08b-6b9b603f370f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4" Oct 30 02:19:09.346669 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.346626 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446df54f-cdf8-4604-a08b-6b9b603f370f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-400b9185-dhzc4\" (UID: \"446df54f-cdf8-4604-a08b-6b9b603f370f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4" Oct 30 02:19:09.348281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.348257 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446df54f-cdf8-4604-a08b-6b9b603f370f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-400b9185-dhzc4\" (UID: \"446df54f-cdf8-4604-a08b-6b9b603f370f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4" Oct 30 02:19:09.367908 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.367880 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhqsc\" (UniqueName: \"kubernetes.io/projected/446df54f-cdf8-4604-a08b-6b9b603f370f-kube-api-access-xhqsc\") pod \"maas-api-revoke-sub-400b9185-dhzc4\" (UID: \"446df54f-cdf8-4604-a08b-6b9b603f370f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4" Oct 30 02:19:09.492522 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.492405 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4" Oct 30 02:19:09.804000 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:09.803951 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4"] Oct 30 02:19:09.807141 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:19:09.807111 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446df54f_cdf8_4604_a08b_6b9b603f370f.slice/crio-897995a2c91632268a1b88f402cf54cd994670f0802b0fade85f36e8b94e3c64 WatchSource:0}: Error finding container 897995a2c91632268a1b88f402cf54cd994670f0802b0fade85f36e8b94e3c64: Status 404 returned error can't find the container with id 897995a2c91632268a1b88f402cf54cd994670f0802b0fade85f36e8b94e3c64 Oct 30 02:19:10.448634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:10.448601 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:10.642548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:10.642456 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4" event={"ID":"446df54f-cdf8-4604-a08b-6b9b603f370f","Type":"ContainerStarted","Data":"d778228816e74ab67a4b0fdfd521b4ae0a5673179c0219e57f2a31932cf96f3f"} Oct 30 02:19:10.642548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:10.642512 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4" event={"ID":"446df54f-cdf8-4604-a08b-6b9b603f370f","Type":"ContainerStarted","Data":"897995a2c91632268a1b88f402cf54cd994670f0802b0fade85f36e8b94e3c64"} Oct 30 02:19:10.683002 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:10.682946 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4" podStartSLOduration=1.682924874 podStartE2EDuration="1.682924874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:19:10.679662209 +0000 UTC m=+2980.258453291" watchObservedRunningTime="2026-10-30 02:19:10.682924874 +0000 UTC m=+2980.261715954" Oct 30 02:19:11.651529 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:11.651497 5998 generic.go:356] "Generic (PLEG): container finished" podID="446df54f-cdf8-4604-a08b-6b9b603f370f" containerID="d778228816e74ab67a4b0fdfd521b4ae0a5673179c0219e57f2a31932cf96f3f" exitCode=0 Oct 30 02:19:11.651976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:11.651572 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4" event={"ID":"446df54f-cdf8-4604-a08b-6b9b603f370f","Type":"ContainerDied","Data":"d778228816e74ab67a4b0fdfd521b4ae0a5673179c0219e57f2a31932cf96f3f"} Oct 30 02:19:11.972463 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:11.972428 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:12.825192 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:12.825163 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4" Oct 30 02:19:12.896302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:12.896265 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/446df54f-cdf8-4604-a08b-6b9b603f370f-kube-api-access-xhqsc\" (UniqueName: \"kubernetes.io/projected/446df54f-cdf8-4604-a08b-6b9b603f370f-kube-api-access-xhqsc\") pod \"446df54f-cdf8-4604-a08b-6b9b603f370f\" (UID: \"446df54f-cdf8-4604-a08b-6b9b603f370f\") " Oct 30 02:19:12.896476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:12.896310 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/446df54f-cdf8-4604-a08b-6b9b603f370f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446df54f-cdf8-4604-a08b-6b9b603f370f-maas-api-openshift-ca\") pod \"446df54f-cdf8-4604-a08b-6b9b603f370f\" (UID: \"446df54f-cdf8-4604-a08b-6b9b603f370f\") " Oct 30 02:19:12.896476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:12.896354 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/446df54f-cdf8-4604-a08b-6b9b603f370f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446df54f-cdf8-4604-a08b-6b9b603f370f-maas-api-xks-ca\") pod \"446df54f-cdf8-4604-a08b-6b9b603f370f\" (UID: \"446df54f-cdf8-4604-a08b-6b9b603f370f\") " Oct 30 02:19:12.896741 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:12.896713 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/446df54f-cdf8-4604-a08b-6b9b603f370f-maas-api-openshift-ca" pod "446df54f-cdf8-4604-a08b-6b9b603f370f" (UID: "446df54f-cdf8-4604-a08b-6b9b603f370f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:12.898515 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:12.898484 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/446df54f-cdf8-4604-a08b-6b9b603f370f-maas-api-xks-ca" pod "446df54f-cdf8-4604-a08b-6b9b603f370f" (UID: "446df54f-cdf8-4604-a08b-6b9b603f370f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:12.898747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:12.898715 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446df54f-cdf8-4604-a08b-6b9b603f370f-kube-api-access-xhqsc" pod "446df54f-cdf8-4604-a08b-6b9b603f370f" (UID: "446df54f-cdf8-4604-a08b-6b9b603f370f"). InnerVolumeSpecName "kube-api-access-xhqsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:12.997908 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:12.997872 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhqsc\" (UniqueName: \"kubernetes.io/projected/446df54f-cdf8-4604-a08b-6b9b603f370f-kube-api-access-xhqsc\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:12.997908 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:12.997903 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/446df54f-cdf8-4604-a08b-6b9b603f370f-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:12.997908 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:12.997914 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/446df54f-cdf8-4604-a08b-6b9b603f370f-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:13.658228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:13.658169 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:13.673415 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:13.673382 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4" Oct 30 02:19:13.673415 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:13.673404 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4" event={"ID":"446df54f-cdf8-4604-a08b-6b9b603f370f","Type":"ContainerDied","Data":"897995a2c91632268a1b88f402cf54cd994670f0802b0fade85f36e8b94e3c64"} Oct 30 02:19:13.673622 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:13.673450 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="897995a2c91632268a1b88f402cf54cd994670f0802b0fade85f36e8b94e3c64" Oct 30 02:19:15.139423 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:15.139382 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:16.698901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:16.698865 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:18.271343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:18.271305 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:19.857838 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:19.857798 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:20.405807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:20.405772 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:20.591841 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:20.591800 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:20.727106 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:20.727057 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:21.101508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:21.101425 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:21.236164 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:21.236107 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:21.393161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:21.393026 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:21.561809 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:21.561760 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:21.727826 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:21.727779 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:21.861237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:21.861202 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:22.453483 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:22.453447 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:22.565349 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:22.565317 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:23.674015 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:23.673976 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:23.780137 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:23.780100 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:24.052436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.052358 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s"] Oct 30 02:19:24.052946 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.052932 5998 cpu_manager.go:414] "containerMap: removing container" podUID="446df54f-cdf8-4604-a08b-6b9b603f370f" containerName="revoke-keys" Oct 30 02:19:24.052996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.052948 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="446df54f-cdf8-4604-a08b-6b9b603f370f" containerName="revoke-keys" Oct 30 02:19:24.053060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.053051 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="446df54f-cdf8-4604-a08b-6b9b603f370f" containerName="revoke-keys" Oct 30 02:19:24.056731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.056713 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" Oct 30 02:19:24.071776 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.071754 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"agc-cred-5d3c5123e53d\"" Oct 30 02:19:24.074988 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.074972 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"routing-overlay\"" Oct 30 02:19:24.082360 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.082338 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"kube-root-ca.crt\"" Oct 30 02:19:24.088316 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.088291 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"payload-processing-external-model-plugins-e2e-extmod-a-58fad4\"" Oct 30 02:19:24.088437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.088334 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"openshift-service-ca.crt\"" Oct 30 02:19:24.089529 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.089505 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/428ef227-997d-4210-a1ef-de5cf92fc1a5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s\" (UID: \"428ef227-997d-4210-a1ef-de5cf92fc1a5\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" Oct 30 02:19:24.089631 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.089556 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428ef227-997d-4210-a1ef-de5cf92fc1a5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s\" (UID: \"428ef227-997d-4210-a1ef-de5cf92fc1a5\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" Oct 30 02:19:24.089631 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.089603 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kfjp\" (UniqueName: \"kubernetes.io/projected/428ef227-997d-4210-a1ef-de5cf92fc1a5-kube-api-access-7kfjp\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s\" (UID: \"428ef227-997d-4210-a1ef-de5cf92fc1a5\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" Oct 30 02:19:24.089737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.089708 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428ef227-997d-4210-a1ef-de5cf92fc1a5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s\" (UID: \"428ef227-997d-4210-a1ef-de5cf92fc1a5\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" Oct 30 02:19:24.094119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.094081 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s"] Oct 30 02:19:24.206115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.193389 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/428ef227-997d-4210-a1ef-de5cf92fc1a5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s\" (UID: \"428ef227-997d-4210-a1ef-de5cf92fc1a5\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" Oct 30 02:19:24.206115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.193457 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428ef227-997d-4210-a1ef-de5cf92fc1a5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s\" (UID: \"428ef227-997d-4210-a1ef-de5cf92fc1a5\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" Oct 30 02:19:24.206115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.193504 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kfjp\" (UniqueName: \"kubernetes.io/projected/428ef227-997d-4210-a1ef-de5cf92fc1a5-kube-api-access-7kfjp\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s\" (UID: \"428ef227-997d-4210-a1ef-de5cf92fc1a5\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" Oct 30 02:19:24.206115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.193621 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428ef227-997d-4210-a1ef-de5cf92fc1a5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s\" (UID: \"428ef227-997d-4210-a1ef-de5cf92fc1a5\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" Oct 30 02:19:24.226114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.219073 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428ef227-997d-4210-a1ef-de5cf92fc1a5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s\" (UID: \"428ef227-997d-4210-a1ef-de5cf92fc1a5\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" Oct 30 02:19:24.280123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.260694 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428ef227-997d-4210-a1ef-de5cf92fc1a5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s\" (UID: \"428ef227-997d-4210-a1ef-de5cf92fc1a5\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" Oct 30 02:19:24.280123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.272900 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-58fad4\"/\"payload-processing-external-model-e2e--4b9246f1-dockercfg-jv5rk\"" Oct 30 02:19:24.300119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.280853 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/428ef227-997d-4210-a1ef-de5cf92fc1a5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s\" (UID: \"428ef227-997d-4210-a1ef-de5cf92fc1a5\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" Oct 30 02:19:24.322163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.314611 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kfjp\" (UniqueName: \"kubernetes.io/projected/428ef227-997d-4210-a1ef-de5cf92fc1a5-kube-api-access-7kfjp\") pod \"payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s\" (UID: \"428ef227-997d-4210-a1ef-de5cf92fc1a5\") " pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" Oct 30 02:19:24.370400 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.370370 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" Oct 30 02:19:24.678734 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.678697 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s"] Oct 30 02:19:24.682182 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:19:24.682147 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428ef227_997d_4210_a1ef_de5cf92fc1a5.slice/crio-ec5bc21879aea8a5dda2d8c6da024ae6195a16bfec57b9c581206bf47ed9399f WatchSource:0}: Error finding container ec5bc21879aea8a5dda2d8c6da024ae6195a16bfec57b9c581206bf47ed9399f: Status 404 returned error can't find the container with id ec5bc21879aea8a5dda2d8c6da024ae6195a16bfec57b9c581206bf47ed9399f Oct 30 02:19:24.795705 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.795673 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" event={"ID":"428ef227-997d-4210-a1ef-de5cf92fc1a5","Type":"ContainerStarted","Data":"bd0d0865e11411c8f0691e152e157874560fc23aeb0da57b670e8dc01603bf6c"} Oct 30 02:19:24.795889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.795726 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" event={"ID":"428ef227-997d-4210-a1ef-de5cf92fc1a5","Type":"ContainerStarted","Data":"ec5bc21879aea8a5dda2d8c6da024ae6195a16bfec57b9c581206bf47ed9399f"} Oct 30 02:19:24.795889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.795851 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" Oct 30 02:19:24.840064 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:24.840012 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" podStartSLOduration=0.839997089 podStartE2EDuration="839.997089ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:19:24.836680012 +0000 UTC m=+2994.415471093" watchObservedRunningTime="2026-10-30 02:19:24.839997089 +0000 UTC m=+2994.418788169" Oct 30 02:19:25.228313 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:19:25.228268 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:19:25.414652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:25.414614 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:26.970207 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:26.970167 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:27.976625 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:27.976586 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:19:28.550702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:28.550664 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:30.293162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:30.293127 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:30.868671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:30.868628 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm"] Oct 30 02:19:30.873785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:30.873752 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm" Oct 30 02:19:30.887426 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:30.887397 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:19:30.898974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:30.898263 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:19:30.898974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:30.898337 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:19:30.898974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:30.898373 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:19:30.907827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:30.904934 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm"] Oct 30 02:19:30.964988 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:30.964944 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltgj2\" (UniqueName: \"kubernetes.io/projected/59c0c20b-4385-42e0-83e9-bdb700eee0b3-kube-api-access-ltgj2\") pod \"maas-api-revoke-sub-afdfb4f7-nbklm\" (UID: \"59c0c20b-4385-42e0-83e9-bdb700eee0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm" Oct 30 02:19:30.965194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:30.965063 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c0c20b-4385-42e0-83e9-bdb700eee0b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afdfb4f7-nbklm\" (UID: \"59c0c20b-4385-42e0-83e9-bdb700eee0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm" Oct 30 02:19:30.965194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:30.965148 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c0c20b-4385-42e0-83e9-bdb700eee0b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afdfb4f7-nbklm\" (UID: \"59c0c20b-4385-42e0-83e9-bdb700eee0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm" Oct 30 02:19:31.067021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:31.066984 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltgj2\" (UniqueName: \"kubernetes.io/projected/59c0c20b-4385-42e0-83e9-bdb700eee0b3-kube-api-access-ltgj2\") pod \"maas-api-revoke-sub-afdfb4f7-nbklm\" (UID: \"59c0c20b-4385-42e0-83e9-bdb700eee0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm" Oct 30 02:19:31.067216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:31.067068 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c0c20b-4385-42e0-83e9-bdb700eee0b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afdfb4f7-nbklm\" (UID: \"59c0c20b-4385-42e0-83e9-bdb700eee0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm" Oct 30 02:19:31.067216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:31.067131 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c0c20b-4385-42e0-83e9-bdb700eee0b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afdfb4f7-nbklm\" (UID: \"59c0c20b-4385-42e0-83e9-bdb700eee0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm" Oct 30 02:19:31.068167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:31.068142 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c0c20b-4385-42e0-83e9-bdb700eee0b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afdfb4f7-nbklm\" (UID: \"59c0c20b-4385-42e0-83e9-bdb700eee0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm" Oct 30 02:19:31.069789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:31.069768 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c0c20b-4385-42e0-83e9-bdb700eee0b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afdfb4f7-nbklm\" (UID: \"59c0c20b-4385-42e0-83e9-bdb700eee0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm" Oct 30 02:19:31.088774 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:31.088743 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltgj2\" (UniqueName: \"kubernetes.io/projected/59c0c20b-4385-42e0-83e9-bdb700eee0b3-kube-api-access-ltgj2\") pod \"maas-api-revoke-sub-afdfb4f7-nbklm\" (UID: \"59c0c20b-4385-42e0-83e9-bdb700eee0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm" Oct 30 02:19:31.199052 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:31.199013 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm" Oct 30 02:19:31.406303 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:31.406269 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:31.550293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:31.549672 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:31.622861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:31.615553 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm"] Oct 30 02:19:31.877719 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:31.877616 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm" event={"ID":"59c0c20b-4385-42e0-83e9-bdb700eee0b3","Type":"ContainerStarted","Data":"c3a8d41df977659276bb779652a6eb34b9d7631ec3b9f07e864420245ca8ac2c"} Oct 30 02:19:32.276624 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.276567 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:32.638595 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.638556 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7"] Oct 30 02:19:32.642963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.642932 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" Oct 30 02:19:32.655856 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.655815 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"agc-cred-eba0f971cd7a\"" Oct 30 02:19:32.666393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.666368 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"payload-processing-external-model-plugins-e2e-extmod-b-000edd\"" Oct 30 02:19:32.666572 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.666375 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"kube-root-ca.crt\"" Oct 30 02:19:32.666649 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.666429 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"openshift-service-ca.crt\"" Oct 30 02:19:32.666649 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.666441 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"agc-cred-822de3ce3685\"" Oct 30 02:19:32.666649 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.666463 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"payload-processing-external-model-e2e--74e737ad-dockercfg-bs8rz\"" Oct 30 02:19:32.666649 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.666475 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-000edd\"/\"routing-overlay\"" Oct 30 02:19:32.670756 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.670735 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7"] Oct 30 02:19:32.679775 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.679746 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec619a8d-98ea-443c-a8b1-3ea4628805f3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7\" (UID: \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" Oct 30 02:19:32.679902 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.679817 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec619a8d-98ea-443c-a8b1-3ea4628805f3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7\" (UID: \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" Oct 30 02:19:32.680041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.680015 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbnff\" (UniqueName: \"kubernetes.io/projected/ec619a8d-98ea-443c-a8b1-3ea4628805f3-kube-api-access-dbnff\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7\" (UID: \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" Oct 30 02:19:32.680219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.680194 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec619a8d-98ea-443c-a8b1-3ea4628805f3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7\" (UID: \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" Oct 30 02:19:32.781833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.781791 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbnff\" (UniqueName: \"kubernetes.io/projected/ec619a8d-98ea-443c-a8b1-3ea4628805f3-kube-api-access-dbnff\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7\" (UID: \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" Oct 30 02:19:32.782016 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.781879 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec619a8d-98ea-443c-a8b1-3ea4628805f3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7\" (UID: \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" Oct 30 02:19:32.782016 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.781954 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec619a8d-98ea-443c-a8b1-3ea4628805f3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7\" (UID: \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" Oct 30 02:19:32.782016 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.781992 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec619a8d-98ea-443c-a8b1-3ea4628805f3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7\" (UID: \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" Oct 30 02:19:32.782825 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.782802 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec619a8d-98ea-443c-a8b1-3ea4628805f3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7\" (UID: \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" Oct 30 02:19:32.782909 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.782834 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec619a8d-98ea-443c-a8b1-3ea4628805f3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7\" (UID: \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" Oct 30 02:19:32.784492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.784475 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec619a8d-98ea-443c-a8b1-3ea4628805f3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7\" (UID: \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" Oct 30 02:19:32.831112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.830606 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbnff\" (UniqueName: \"kubernetes.io/projected/ec619a8d-98ea-443c-a8b1-3ea4628805f3-kube-api-access-dbnff\") pod \"payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7\" (UID: \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\") " pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" Oct 30 02:19:32.886565 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.886522 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm" event={"ID":"59c0c20b-4385-42e0-83e9-bdb700eee0b3","Type":"ContainerStarted","Data":"4509bd21b5b7eec3f48cb5fb12427b795f57049b9e3f806d5ae34ba74a7166c2"} Oct 30 02:19:32.935186 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.933362 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm" podStartSLOduration=2.933339973 podStartE2EDuration="2.933339973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:19:32.931792239 +0000 UTC m=+3002.510583321" watchObservedRunningTime="2026-10-30 02:19:32.933339973 +0000 UTC m=+3002.512131054" Oct 30 02:19:32.957060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:32.957022 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" Oct 30 02:19:33.115582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.102348 5998 scope.go:122] "RemoveContainer" containerID="7e46698ecad9bee81bff804756d76cdd723e9a95be93d6b79264fe6c16f8bdac" Oct 30 02:19:33.154849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.154707 5998 scope.go:122] "RemoveContainer" containerID="dc64e9048efdf74cf60b5425f79299ef9d2312eb3f7e5567b307b595778531d5" Oct 30 02:19:33.164744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.164715 5998 scope.go:122] "RemoveContainer" containerID="3c24b3088408dffbcad7e6cb0d3a2488c057525796291d43b1b3a267b2cba981" Oct 30 02:19:33.176264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.176238 5998 scope.go:122] "RemoveContainer" containerID="c3998c8ccf99ddb02270d84cd65daeb1e2ba44590b354878a011a401e15cfaf8" Oct 30 02:19:33.186607 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.186587 5998 scope.go:122] "RemoveContainer" containerID="f45ac764aab3c576a8905862416bf6206ecc7247be3ad1c9970d967659fd6bd8" Oct 30 02:19:33.338238 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.338166 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7"] Oct 30 02:19:33.343026 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:19:33.341850 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec619a8d_98ea_443c_a8b1_3ea4628805f3.slice/crio-6b4f178a8eaf074fbf9f2e4db202711b8ea694de3d2c14e50a7ce79543b5bd0e WatchSource:0}: Error finding container 6b4f178a8eaf074fbf9f2e4db202711b8ea694de3d2c14e50a7ce79543b5bd0e: Status 404 returned error can't find the container with id 6b4f178a8eaf074fbf9f2e4db202711b8ea694de3d2c14e50a7ce79543b5bd0e Oct 30 02:19:33.457185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.455527 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:33.737237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.736449 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p"] Oct 30 02:19:33.741623 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.741599 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" Oct 30 02:19:33.750300 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.750264 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p"] Oct 30 02:19:33.758675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.755393 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"agc-cred-c7ef5cce95a7\"" Oct 30 02:19:33.758675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.755493 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"openshift-service-ca.crt\"" Oct 30 02:19:33.758675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.755600 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"payload-processing-external-model-e2e--c1fccfb3-dockercfg-zxgs5\"" Oct 30 02:19:33.758675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.755780 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"routing-overlay\"" Oct 30 02:19:33.758675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.755852 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0ce950\"" Oct 30 02:19:33.763009 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.755387 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"kube-root-ca.crt\"" Oct 30 02:19:33.797922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.797879 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c156f00-37ed-4828-a0b6-19c3c985dfd2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p\" (UID: \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" Oct 30 02:19:33.798125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.797932 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c156f00-37ed-4828-a0b6-19c3c985dfd2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p\" (UID: \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" Oct 30 02:19:33.798241 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.798139 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksphm\" (UniqueName: \"kubernetes.io/projected/6c156f00-37ed-4828-a0b6-19c3c985dfd2-kube-api-access-ksphm\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p\" (UID: \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" Oct 30 02:19:33.798354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.798340 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c156f00-37ed-4828-a0b6-19c3c985dfd2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p\" (UID: \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" Oct 30 02:19:33.941337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.924669 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c156f00-37ed-4828-a0b6-19c3c985dfd2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p\" (UID: \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" Oct 30 02:19:33.941337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.924753 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c156f00-37ed-4828-a0b6-19c3c985dfd2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p\" (UID: \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" Oct 30 02:19:33.941337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.924781 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c156f00-37ed-4828-a0b6-19c3c985dfd2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p\" (UID: \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" Oct 30 02:19:33.941337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.924898 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksphm\" (UniqueName: \"kubernetes.io/projected/6c156f00-37ed-4828-a0b6-19c3c985dfd2-kube-api-access-ksphm\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p\" (UID: \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" Oct 30 02:19:33.941337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.935054 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c156f00-37ed-4828-a0b6-19c3c985dfd2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p\" (UID: \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" Oct 30 02:19:33.941337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.935534 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c156f00-37ed-4828-a0b6-19c3c985dfd2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p\" (UID: \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" Oct 30 02:19:33.962235 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:33.961690 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c156f00-37ed-4828-a0b6-19c3c985dfd2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p\" (UID: \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" Oct 30 02:19:34.005167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:34.001232 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:34.005167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:34.003242 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksphm\" (UniqueName: \"kubernetes.io/projected/6c156f00-37ed-4828-a0b6-19c3c985dfd2-kube-api-access-ksphm\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p\" (UID: \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" Oct 30 02:19:34.021346 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:34.021290 5998 generic.go:356] "Generic (PLEG): container finished" podID="59c0c20b-4385-42e0-83e9-bdb700eee0b3" containerID="4509bd21b5b7eec3f48cb5fb12427b795f57049b9e3f806d5ae34ba74a7166c2" exitCode=0 Oct 30 02:19:34.021706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:34.021665 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm" event={"ID":"59c0c20b-4385-42e0-83e9-bdb700eee0b3","Type":"ContainerDied","Data":"4509bd21b5b7eec3f48cb5fb12427b795f57049b9e3f806d5ae34ba74a7166c2"} Oct 30 02:19:34.024904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:34.024869 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" event={"ID":"ec619a8d-98ea-443c-a8b1-3ea4628805f3","Type":"ContainerStarted","Data":"49c33a7403ca1c38749bbd22f3950824153112de9948921beef267205e421738"} Oct 30 02:19:34.033556 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:34.033517 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" event={"ID":"ec619a8d-98ea-443c-a8b1-3ea4628805f3","Type":"ContainerStarted","Data":"6b4f178a8eaf074fbf9f2e4db202711b8ea694de3d2c14e50a7ce79543b5bd0e"} Oct 30 02:19:34.034669 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:34.034642 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" Oct 30 02:19:34.072113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:34.072003 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" Oct 30 02:19:34.129338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:34.129247 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" podStartSLOduration=2.129226468 podStartE2EDuration="2.129226468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:19:34.116431226 +0000 UTC m=+3003.695222306" watchObservedRunningTime="2026-10-30 02:19:34.129226468 +0000 UTC m=+3003.708017548" Oct 30 02:19:34.479998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:34.479964 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p"] Oct 30 02:19:34.483166 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:19:34.483137 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c156f00_37ed_4828_a0b6_19c3c985dfd2.slice/crio-790a68bc692d351e3f7fc90d28bb96de31e89f7f887f2b3b162d9bcc4dedf448 WatchSource:0}: Error finding container 790a68bc692d351e3f7fc90d28bb96de31e89f7f887f2b3b162d9bcc4dedf448: Status 404 returned error can't find the container with id 790a68bc692d351e3f7fc90d28bb96de31e89f7f887f2b3b162d9bcc4dedf448 Oct 30 02:19:35.040911 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:35.040874 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" event={"ID":"6c156f00-37ed-4828-a0b6-19c3c985dfd2","Type":"ContainerStarted","Data":"08f6160209df1d9f8a3766c16cc5af15ae19e41bc47aa1c1d52df41f0357f8ca"} Oct 30 02:19:35.041458 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:35.040939 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" event={"ID":"6c156f00-37ed-4828-a0b6-19c3c985dfd2","Type":"ContainerStarted","Data":"790a68bc692d351e3f7fc90d28bb96de31e89f7f887f2b3b162d9bcc4dedf448"} Oct 30 02:19:35.041458 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:35.041119 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" Oct 30 02:19:35.082596 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:35.082533 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" podStartSLOduration=2.082514565 podStartE2EDuration="2.082514565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:19:35.078652123 +0000 UTC m=+3004.657443203" watchObservedRunningTime="2026-10-30 02:19:35.082514565 +0000 UTC m=+3004.661305645" Oct 30 02:19:35.317180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:35.317140 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm" Oct 30 02:19:35.464759 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:35.464722 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c0c20b-4385-42e0-83e9-bdb700eee0b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c0c20b-4385-42e0-83e9-bdb700eee0b3-maas-api-openshift-ca\") pod \"59c0c20b-4385-42e0-83e9-bdb700eee0b3\" (UID: \"59c0c20b-4385-42e0-83e9-bdb700eee0b3\") " Oct 30 02:19:35.464958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:35.464846 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c0c20b-4385-42e0-83e9-bdb700eee0b3-kube-api-access-ltgj2\" (UniqueName: \"kubernetes.io/projected/59c0c20b-4385-42e0-83e9-bdb700eee0b3-kube-api-access-ltgj2\") pod \"59c0c20b-4385-42e0-83e9-bdb700eee0b3\" (UID: \"59c0c20b-4385-42e0-83e9-bdb700eee0b3\") " Oct 30 02:19:35.464958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:35.464925 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59c0c20b-4385-42e0-83e9-bdb700eee0b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c0c20b-4385-42e0-83e9-bdb700eee0b3-maas-api-xks-ca\") pod \"59c0c20b-4385-42e0-83e9-bdb700eee0b3\" (UID: \"59c0c20b-4385-42e0-83e9-bdb700eee0b3\") " Oct 30 02:19:35.465217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:35.465179 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c0c20b-4385-42e0-83e9-bdb700eee0b3-maas-api-openshift-ca" pod "59c0c20b-4385-42e0-83e9-bdb700eee0b3" (UID: "59c0c20b-4385-42e0-83e9-bdb700eee0b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:35.465555 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:35.465533 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c0c20b-4385-42e0-83e9-bdb700eee0b3-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.467077 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:35.467049 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c0c20b-4385-42e0-83e9-bdb700eee0b3-maas-api-xks-ca" pod "59c0c20b-4385-42e0-83e9-bdb700eee0b3" (UID: "59c0c20b-4385-42e0-83e9-bdb700eee0b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:35.467353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:35.467331 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c0c20b-4385-42e0-83e9-bdb700eee0b3-kube-api-access-ltgj2" pod "59c0c20b-4385-42e0-83e9-bdb700eee0b3" (UID: "59c0c20b-4385-42e0-83e9-bdb700eee0b3"). InnerVolumeSpecName "kube-api-access-ltgj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:35.567641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:35.567556 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltgj2\" (UniqueName: \"kubernetes.io/projected/59c0c20b-4385-42e0-83e9-bdb700eee0b3-kube-api-access-ltgj2\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.567641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:35.567596 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c0c20b-4385-42e0-83e9-bdb700eee0b3-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.679551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:35.679517 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:35.808195 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:35.808165 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" Oct 30 02:19:36.051585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:36.051550 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm" Oct 30 02:19:36.052047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:36.051580 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm" event={"ID":"59c0c20b-4385-42e0-83e9-bdb700eee0b3","Type":"ContainerDied","Data":"c3a8d41df977659276bb779652a6eb34b9d7631ec3b9f07e864420245ca8ac2c"} Oct 30 02:19:36.052047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:36.051640 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3a8d41df977659276bb779652a6eb34b9d7631ec3b9f07e864420245ca8ac2c" Oct 30 02:19:37.182965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:37.182924 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:38.099839 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:38.099806 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:38.625272 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:38.620169 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p"] Oct 30 02:19:38.691065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:38.690914 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:38.795722 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:38.795685 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:38.901296 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:38.901183 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:39.885476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:39.885440 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7"] Oct 30 02:19:39.888385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:39.888358 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" Oct 30 02:19:40.122605 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:40.122570 5998 generic.go:356] "Generic (PLEG): container finished" podID="7076db80-f9a3-423f-b198-f0a7aba80bd7" containerID="912eab5621d876448ae3aa2a0207002feac417c12e21aa8cae2226c48ff37021" exitCode=0 Oct 30 02:19:40.122790 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:40.122624 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" event={"ID":"7076db80-f9a3-423f-b198-f0a7aba80bd7","Type":"ContainerDied","Data":"912eab5621d876448ae3aa2a0207002feac417c12e21aa8cae2226c48ff37021"} Oct 30 02:19:40.123079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:40.123064 5998 scope.go:122] "RemoveContainer" containerID="912eab5621d876448ae3aa2a0207002feac417c12e21aa8cae2226c48ff37021" Oct 30 02:19:40.124989 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:40.124955 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:19:40.125124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:40.125028 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:19:40.125124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:40.125060 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:19:40.595760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:40.595714 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:40.911404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:40.911366 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:41.031111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:41.027686 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:41.147042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:41.146830 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" event={"ID":"7076db80-f9a3-423f-b198-f0a7aba80bd7","Type":"ContainerStarted","Data":"99e92e7f5539f677d3f2780853aa31a4e19ea6f62559d08218f1717923b078d3"} Oct 30 02:19:41.438539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:41.438491 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:19:41.440013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:41.439982 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 02:19:41.440112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:41.440047 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" podUID="7076db80-f9a3-423f-b198-f0a7aba80bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 02:19:41.728243 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:41.720589 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:41.837111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:41.833654 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:42.125162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.119219 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:42.236116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.230199 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:42.307118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.301877 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:19:42.450116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.442630 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 02:19:42.450116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.442692 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" podUID="7076db80-f9a3-423f-b198-f0a7aba80bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 02:19:42.491297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.491153 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:42.493154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.493131 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr"] Oct 30 02:19:42.494237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.494212 5998 cpu_manager.go:414] "containerMap: removing container" podUID="59c0c20b-4385-42e0-83e9-bdb700eee0b3" containerName="revoke-keys" Oct 30 02:19:42.494401 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.494375 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c0c20b-4385-42e0-83e9-bdb700eee0b3" containerName="revoke-keys" Oct 30 02:19:42.494746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.494728 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c0c20b-4385-42e0-83e9-bdb700eee0b3" containerName="revoke-keys" Oct 30 02:19:42.498560 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.498533 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" Oct 30 02:19:42.513925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.513891 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr"] Oct 30 02:19:42.543504 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.543471 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"agc-cred-437db4025c2d\"" Oct 30 02:19:42.551851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.544610 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"openshift-service-ca.crt\"" Oct 30 02:19:42.551851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.547415 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc2f4f37-9a00-420f-9020-d6c74b917861-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr\" (UID: \"dc2f4f37-9a00-420f-9020-d6c74b917861\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" Oct 30 02:19:42.551851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.550036 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc2f4f37-9a00-420f-9020-d6c74b917861-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr\" (UID: \"dc2f4f37-9a00-420f-9020-d6c74b917861\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" Oct 30 02:19:42.551851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.547984 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"kube-root-ca.crt\"" Oct 30 02:19:42.551851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.550265 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc2f4f37-9a00-420f-9020-d6c74b917861-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr\" (UID: \"dc2f4f37-9a00-420f-9020-d6c74b917861\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" Oct 30 02:19:42.551851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.549524 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"payload-processing-external-model-e2e--1ed6558d-dockercfg-xn4z5\"" Oct 30 02:19:42.551851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.548486 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-9dfb3c\"" Oct 30 02:19:42.551851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.550408 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scxm8\" (UniqueName: \"kubernetes.io/projected/dc2f4f37-9a00-420f-9020-d6c74b917861-kube-api-access-scxm8\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr\" (UID: \"dc2f4f37-9a00-420f-9020-d6c74b917861\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" Oct 30 02:19:42.551851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.549575 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"agc-cred-1991e0bfc294\"" Oct 30 02:19:42.555814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.555790 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9dfb3c\"/\"routing-overlay\"" Oct 30 02:19:42.653143 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.653102 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scxm8\" (UniqueName: \"kubernetes.io/projected/dc2f4f37-9a00-420f-9020-d6c74b917861-kube-api-access-scxm8\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr\" (UID: \"dc2f4f37-9a00-420f-9020-d6c74b917861\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" Oct 30 02:19:42.653360 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.653210 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc2f4f37-9a00-420f-9020-d6c74b917861-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr\" (UID: \"dc2f4f37-9a00-420f-9020-d6c74b917861\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" Oct 30 02:19:42.653360 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.653321 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc2f4f37-9a00-420f-9020-d6c74b917861-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr\" (UID: \"dc2f4f37-9a00-420f-9020-d6c74b917861\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" Oct 30 02:19:42.653490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.653365 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc2f4f37-9a00-420f-9020-d6c74b917861-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr\" (UID: \"dc2f4f37-9a00-420f-9020-d6c74b917861\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" Oct 30 02:19:42.653966 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.653929 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc2f4f37-9a00-420f-9020-d6c74b917861-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr\" (UID: \"dc2f4f37-9a00-420f-9020-d6c74b917861\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" Oct 30 02:19:42.654158 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.654068 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc2f4f37-9a00-420f-9020-d6c74b917861-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr\" (UID: \"dc2f4f37-9a00-420f-9020-d6c74b917861\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" Oct 30 02:19:42.655955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.655930 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc2f4f37-9a00-420f-9020-d6c74b917861-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr\" (UID: \"dc2f4f37-9a00-420f-9020-d6c74b917861\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" Oct 30 02:19:42.676515 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.676481 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scxm8\" (UniqueName: \"kubernetes.io/projected/dc2f4f37-9a00-420f-9020-d6c74b917861-kube-api-access-scxm8\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr\" (UID: \"dc2f4f37-9a00-420f-9020-d6c74b917861\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" Oct 30 02:19:42.776168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.775143 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh"] Oct 30 02:19:42.828360 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.828317 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh"] Oct 30 02:19:42.828766 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.828679 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh" Oct 30 02:19:42.959081 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.881991 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgll7\" (UniqueName: \"kubernetes.io/projected/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-kube-api-access-kgll7\") pod \"maas-api-revoke-sub-a096a0d9-dwpqh\" (UID: \"1b1225a9-cd64-4a55-9051-a7fa3a16b3e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh" Oct 30 02:19:42.959081 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.959003 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a096a0d9-dwpqh\" (UID: \"1b1225a9-cd64-4a55-9051-a7fa3a16b3e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh" Oct 30 02:19:42.959399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.959218 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a096a0d9-dwpqh\" (UID: \"1b1225a9-cd64-4a55-9051-a7fa3a16b3e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh" Oct 30 02:19:42.965009 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.964324 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" Oct 30 02:19:42.976321 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.975456 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:19:42.976321 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.975647 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:19:42.976321 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.975661 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:19:42.976321 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:42.975762 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:19:43.060692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.060601 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a096a0d9-dwpqh\" (UID: \"1b1225a9-cd64-4a55-9051-a7fa3a16b3e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh" Oct 30 02:19:43.060860 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.060747 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgll7\" (UniqueName: \"kubernetes.io/projected/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-kube-api-access-kgll7\") pod \"maas-api-revoke-sub-a096a0d9-dwpqh\" (UID: \"1b1225a9-cd64-4a55-9051-a7fa3a16b3e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh" Oct 30 02:19:43.060928 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.060905 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a096a0d9-dwpqh\" (UID: \"1b1225a9-cd64-4a55-9051-a7fa3a16b3e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh" Oct 30 02:19:43.061655 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.061610 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a096a0d9-dwpqh\" (UID: \"1b1225a9-cd64-4a55-9051-a7fa3a16b3e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh" Oct 30 02:19:43.062968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.062948 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a096a0d9-dwpqh\" (UID: \"1b1225a9-cd64-4a55-9051-a7fa3a16b3e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh" Oct 30 02:19:43.094249 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.092844 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgll7\" (UniqueName: \"kubernetes.io/projected/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-kube-api-access-kgll7\") pod \"maas-api-revoke-sub-a096a0d9-dwpqh\" (UID: \"1b1225a9-cd64-4a55-9051-a7fa3a16b3e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh" Oct 30 02:19:43.176304 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.176056 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx"] Oct 30 02:19:43.182414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.181419 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx" Oct 30 02:19:43.202708 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.202678 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx"] Oct 30 02:19:43.263741 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.263708 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230d086e-bfe5-4c4a-805b-404cd9c5cba7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9d99c15-lh5mx\" (UID: \"230d086e-bfe5-4c4a-805b-404cd9c5cba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx" Oct 30 02:19:43.264062 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.264004 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qljg\" (UniqueName: \"kubernetes.io/projected/230d086e-bfe5-4c4a-805b-404cd9c5cba7-kube-api-access-5qljg\") pod \"maas-api-revoke-sub-d9d99c15-lh5mx\" (UID: \"230d086e-bfe5-4c4a-805b-404cd9c5cba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx" Oct 30 02:19:43.264238 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.264217 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230d086e-bfe5-4c4a-805b-404cd9c5cba7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9d99c15-lh5mx\" (UID: \"230d086e-bfe5-4c4a-805b-404cd9c5cba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx" Oct 30 02:19:43.281942 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.281888 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh" Oct 30 02:19:43.334758 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.334632 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr"] Oct 30 02:19:43.335935 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:19:43.335893 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2f4f37_9a00_420f_9020_d6c74b917861.slice/crio-590b6e6a58a4b3c55499db2c8cc99e9e93b0834e12d89ea7462a2b9dba3cfc4d WatchSource:0}: Error finding container 590b6e6a58a4b3c55499db2c8cc99e9e93b0834e12d89ea7462a2b9dba3cfc4d: Status 404 returned error can't find the container with id 590b6e6a58a4b3c55499db2c8cc99e9e93b0834e12d89ea7462a2b9dba3cfc4d Oct 30 02:19:43.365229 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.365201 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qljg\" (UniqueName: \"kubernetes.io/projected/230d086e-bfe5-4c4a-805b-404cd9c5cba7-kube-api-access-5qljg\") pod \"maas-api-revoke-sub-d9d99c15-lh5mx\" (UID: \"230d086e-bfe5-4c4a-805b-404cd9c5cba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx" Oct 30 02:19:43.365373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.365325 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230d086e-bfe5-4c4a-805b-404cd9c5cba7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9d99c15-lh5mx\" (UID: \"230d086e-bfe5-4c4a-805b-404cd9c5cba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx" Oct 30 02:19:43.365373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.365350 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230d086e-bfe5-4c4a-805b-404cd9c5cba7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9d99c15-lh5mx\" (UID: \"230d086e-bfe5-4c4a-805b-404cd9c5cba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx" Oct 30 02:19:43.366030 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.366004 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230d086e-bfe5-4c4a-805b-404cd9c5cba7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9d99c15-lh5mx\" (UID: \"230d086e-bfe5-4c4a-805b-404cd9c5cba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx" Oct 30 02:19:43.367699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.367674 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230d086e-bfe5-4c4a-805b-404cd9c5cba7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9d99c15-lh5mx\" (UID: \"230d086e-bfe5-4c4a-805b-404cd9c5cba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx" Oct 30 02:19:43.389660 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.389634 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qljg\" (UniqueName: \"kubernetes.io/projected/230d086e-bfe5-4c4a-805b-404cd9c5cba7-kube-api-access-5qljg\") pod \"maas-api-revoke-sub-d9d99c15-lh5mx\" (UID: \"230d086e-bfe5-4c4a-805b-404cd9c5cba7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx" Oct 30 02:19:43.443115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.442513 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 02:19:43.443115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.442567 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" podUID="7076db80-f9a3-423f-b198-f0a7aba80bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 02:19:43.497316 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.497279 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx" Oct 30 02:19:43.621350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.621149 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" podUID="6c156f00-37ed-4828-a0b6-19c3c985dfd2" containerName="payload-processing" containerID="cri-o://08f6160209df1d9f8a3766c16cc5af15ae19e41bc47aa1c1d52df41f0357f8ca" gracePeriod=25 Oct 30 02:19:43.633395 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.633362 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh"] Oct 30 02:19:43.834206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:43.834162 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx"] Oct 30 02:19:43.837266 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:19:43.837227 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230d086e_bfe5_4c4a_805b_404cd9c5cba7.slice/crio-565c03a525444c0e1192ac2b9be52c0bc9d60d5fc64a9ed8fe9c5553d6d42576 WatchSource:0}: Error finding container 565c03a525444c0e1192ac2b9be52c0bc9d60d5fc64a9ed8fe9c5553d6d42576: Status 404 returned error can't find the container with id 565c03a525444c0e1192ac2b9be52c0bc9d60d5fc64a9ed8fe9c5553d6d42576 Oct 30 02:19:44.144612 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.144581 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" Oct 30 02:19:44.168032 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.167996 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:44.176202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.176169 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c156f00-37ed-4828-a0b6-19c3c985dfd2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c156f00-37ed-4828-a0b6-19c3c985dfd2-routing-overlay-volume\") pod \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\" (UID: \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\") " Oct 30 02:19:44.176355 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.176306 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c156f00-37ed-4828-a0b6-19c3c985dfd2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c156f00-37ed-4828-a0b6-19c3c985dfd2-provider-credentials\") pod \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\" (UID: \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\") " Oct 30 02:19:44.176799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.176546 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c156f00-37ed-4828-a0b6-19c3c985dfd2-kube-api-access-ksphm\" (UniqueName: \"kubernetes.io/projected/6c156f00-37ed-4828-a0b6-19c3c985dfd2-kube-api-access-ksphm\") pod \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\" (UID: \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\") " Oct 30 02:19:44.176799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.176589 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c156f00-37ed-4828-a0b6-19c3c985dfd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c156f00-37ed-4828-a0b6-19c3c985dfd2-plugins-config-volume\") pod \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\" (UID: \"6c156f00-37ed-4828-a0b6-19c3c985dfd2\") " Oct 30 02:19:44.176799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.176621 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c156f00-37ed-4828-a0b6-19c3c985dfd2-routing-overlay-volume" pod "6c156f00-37ed-4828-a0b6-19c3c985dfd2" (UID: "6c156f00-37ed-4828-a0b6-19c3c985dfd2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:44.177151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.177128 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c156f00-37ed-4828-a0b6-19c3c985dfd2-plugins-config-volume" pod "6c156f00-37ed-4828-a0b6-19c3c985dfd2" (UID: "6c156f00-37ed-4828-a0b6-19c3c985dfd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:44.177310 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.177288 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c156f00-37ed-4828-a0b6-19c3c985dfd2-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:44.177388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.177316 5998 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c156f00-37ed-4828-a0b6-19c3c985dfd2-routing-overlay-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:44.186309 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.186277 5998 generic.go:356] "Generic (PLEG): container finished" podID="6c156f00-37ed-4828-a0b6-19c3c985dfd2" containerID="08f6160209df1d9f8a3766c16cc5af15ae19e41bc47aa1c1d52df41f0357f8ca" exitCode=0 Oct 30 02:19:44.186512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.186317 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" event={"ID":"6c156f00-37ed-4828-a0b6-19c3c985dfd2","Type":"ContainerDied","Data":"08f6160209df1d9f8a3766c16cc5af15ae19e41bc47aa1c1d52df41f0357f8ca"} Oct 30 02:19:44.186512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.186385 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" event={"ID":"6c156f00-37ed-4828-a0b6-19c3c985dfd2","Type":"ContainerDied","Data":"790a68bc692d351e3f7fc90d28bb96de31e89f7f887f2b3b162d9bcc4dedf448"} Oct 30 02:19:44.186512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.186403 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p" Oct 30 02:19:44.187408 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.186447 5998 scope.go:122] "RemoveContainer" containerID="08f6160209df1d9f8a3766c16cc5af15ae19e41bc47aa1c1d52df41f0357f8ca" Oct 30 02:19:44.188507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.188480 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx" event={"ID":"230d086e-bfe5-4c4a-805b-404cd9c5cba7","Type":"ContainerStarted","Data":"565c03a525444c0e1192ac2b9be52c0bc9d60d5fc64a9ed8fe9c5553d6d42576"} Oct 30 02:19:44.190167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.190139 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh" event={"ID":"1b1225a9-cd64-4a55-9051-a7fa3a16b3e7","Type":"ContainerStarted","Data":"453dfba678f8b663033d2dc536cf9acac127231b33ca68f1916b8258792bcaf3"} Oct 30 02:19:44.191639 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.191617 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" event={"ID":"dc2f4f37-9a00-420f-9020-d6c74b917861","Type":"ContainerStarted","Data":"605c4701997e3ef71474bfea3849ed211f43657cef2e8274df1fdf6754b99512"} Oct 30 02:19:44.191750 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.191661 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" event={"ID":"dc2f4f37-9a00-420f-9020-d6c74b917861","Type":"ContainerStarted","Data":"590b6e6a58a4b3c55499db2c8cc99e9e93b0834e12d89ea7462a2b9dba3cfc4d"} Oct 30 02:19:44.191750 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.191691 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" Oct 30 02:19:44.198900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.198881 5998 scope.go:122] "RemoveContainer" containerID="08f6160209df1d9f8a3766c16cc5af15ae19e41bc47aa1c1d52df41f0357f8ca" Oct 30 02:19:44.199244 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:19:44.199217 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f6160209df1d9f8a3766c16cc5af15ae19e41bc47aa1c1d52df41f0357f8ca\": container with ID starting with 08f6160209df1d9f8a3766c16cc5af15ae19e41bc47aa1c1d52df41f0357f8ca not found: ID does not exist" containerID="08f6160209df1d9f8a3766c16cc5af15ae19e41bc47aa1c1d52df41f0357f8ca" Oct 30 02:19:44.199359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.199256 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f6160209df1d9f8a3766c16cc5af15ae19e41bc47aa1c1d52df41f0357f8ca"} err="failed to get container status \"08f6160209df1d9f8a3766c16cc5af15ae19e41bc47aa1c1d52df41f0357f8ca\": rpc error: code = NotFound desc = could not find container \"08f6160209df1d9f8a3766c16cc5af15ae19e41bc47aa1c1d52df41f0357f8ca\": container with ID starting with 08f6160209df1d9f8a3766c16cc5af15ae19e41bc47aa1c1d52df41f0357f8ca not found: ID does not exist" Oct 30 02:19:44.264866 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.264813 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c156f00-37ed-4828-a0b6-19c3c985dfd2-kube-api-access-ksphm" pod "6c156f00-37ed-4828-a0b6-19c3c985dfd2" (UID: "6c156f00-37ed-4828-a0b6-19c3c985dfd2"). InnerVolumeSpecName "kube-api-access-ksphm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:44.270060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.268250 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c156f00-37ed-4828-a0b6-19c3c985dfd2-provider-credentials" pod "6c156f00-37ed-4828-a0b6-19c3c985dfd2" (UID: "6c156f00-37ed-4828-a0b6-19c3c985dfd2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:44.285343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.283860 5998 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c156f00-37ed-4828-a0b6-19c3c985dfd2-provider-credentials\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:44.285343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.283889 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksphm\" (UniqueName: \"kubernetes.io/projected/6c156f00-37ed-4828-a0b6-19c3c985dfd2-kube-api-access-ksphm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:44.332628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.332558 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" podStartSLOduration=2.332540236 podStartE2EDuration="2.332540236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:19:44.328627419 +0000 UTC m=+3013.907418499" watchObservedRunningTime="2026-10-30 02:19:44.332540236 +0000 UTC m=+3013.911331318" Oct 30 02:19:44.438352 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.438317 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 02:19:44.438554 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.438371 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" podUID="7076db80-f9a3-423f-b198-f0a7aba80bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 02:19:44.542326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.542291 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p"] Oct 30 02:19:44.546592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.546561 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-fc96ksw6p"] Oct 30 02:19:44.886463 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:44.886373 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" podUID="ec619a8d-98ea-443c-a8b1-3ea4628805f3" containerName="payload-processing" containerID="cri-o://49c33a7403ca1c38749bbd22f3950824153112de9948921beef267205e421738" gracePeriod=25 Oct 30 02:19:45.224676 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.224586 5998 generic.go:356] "Generic (PLEG): container finished" podID="230d086e-bfe5-4c4a-805b-404cd9c5cba7" containerID="88bd67b836c034b974a0f3b5ca0901994a64142fdf8d717aabfb64913c80a487" exitCode=0 Oct 30 02:19:45.230418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.229696 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx" event={"ID":"230d086e-bfe5-4c4a-805b-404cd9c5cba7","Type":"ContainerDied","Data":"88bd67b836c034b974a0f3b5ca0901994a64142fdf8d717aabfb64913c80a487"} Oct 30 02:19:45.234548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.234388 5998 generic.go:356] "Generic (PLEG): container finished" podID="ec619a8d-98ea-443c-a8b1-3ea4628805f3" containerID="49c33a7403ca1c38749bbd22f3950824153112de9948921beef267205e421738" exitCode=0 Oct 30 02:19:45.234548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.234485 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" event={"ID":"ec619a8d-98ea-443c-a8b1-3ea4628805f3","Type":"ContainerDied","Data":"49c33a7403ca1c38749bbd22f3950824153112de9948921beef267205e421738"} Oct 30 02:19:45.250119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.244325 5998 generic.go:356] "Generic (PLEG): container finished" podID="1b1225a9-cd64-4a55-9051-a7fa3a16b3e7" containerID="5b8eb5f2e71f7f87cd0f399867a0422d83b4dde6905ed2f47fe2215695d939c4" exitCode=0 Oct 30 02:19:45.250119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.245305 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh" event={"ID":"1b1225a9-cd64-4a55-9051-a7fa3a16b3e7","Type":"ContainerDied","Data":"5b8eb5f2e71f7f87cd0f399867a0422d83b4dde6905ed2f47fe2215695d939c4"} Oct 30 02:19:45.257692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.257668 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" Oct 30 02:19:45.296000 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.295960 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec619a8d-98ea-443c-a8b1-3ea4628805f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec619a8d-98ea-443c-a8b1-3ea4628805f3-plugins-config-volume\") pod \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\" (UID: \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\") " Oct 30 02:19:45.296228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.296023 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec619a8d-98ea-443c-a8b1-3ea4628805f3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec619a8d-98ea-443c-a8b1-3ea4628805f3-provider-credentials\") pod \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\" (UID: \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\") " Oct 30 02:19:45.296228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.296121 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec619a8d-98ea-443c-a8b1-3ea4628805f3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec619a8d-98ea-443c-a8b1-3ea4628805f3-routing-overlay-volume\") pod \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\" (UID: \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\") " Oct 30 02:19:45.296228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.296161 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec619a8d-98ea-443c-a8b1-3ea4628805f3-kube-api-access-dbnff\" (UniqueName: \"kubernetes.io/projected/ec619a8d-98ea-443c-a8b1-3ea4628805f3-kube-api-access-dbnff\") pod \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\" (UID: \"ec619a8d-98ea-443c-a8b1-3ea4628805f3\") " Oct 30 02:19:45.296514 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.296485 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec619a8d-98ea-443c-a8b1-3ea4628805f3-plugins-config-volume" pod "ec619a8d-98ea-443c-a8b1-3ea4628805f3" (UID: "ec619a8d-98ea-443c-a8b1-3ea4628805f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:45.296514 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.296503 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec619a8d-98ea-443c-a8b1-3ea4628805f3-routing-overlay-volume" pod "ec619a8d-98ea-443c-a8b1-3ea4628805f3" (UID: "ec619a8d-98ea-443c-a8b1-3ea4628805f3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:45.297080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.296976 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec619a8d-98ea-443c-a8b1-3ea4628805f3-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:45.297080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.297010 5998 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec619a8d-98ea-443c-a8b1-3ea4628805f3-routing-overlay-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:45.298523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.298502 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec619a8d-98ea-443c-a8b1-3ea4628805f3-kube-api-access-dbnff" pod "ec619a8d-98ea-443c-a8b1-3ea4628805f3" (UID: "ec619a8d-98ea-443c-a8b1-3ea4628805f3"). InnerVolumeSpecName "kube-api-access-dbnff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:45.298599 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.298581 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec619a8d-98ea-443c-a8b1-3ea4628805f3-provider-credentials" pod "ec619a8d-98ea-443c-a8b1-3ea4628805f3" (UID: "ec619a8d-98ea-443c-a8b1-3ea4628805f3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:45.398493 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.398447 5998 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec619a8d-98ea-443c-a8b1-3ea4628805f3-provider-credentials\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:45.398493 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.398487 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbnff\" (UniqueName: \"kubernetes.io/projected/ec619a8d-98ea-443c-a8b1-3ea4628805f3-kube-api-access-dbnff\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:45.438671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.438619 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 02:19:45.438894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.438683 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" podUID="7076db80-f9a3-423f-b198-f0a7aba80bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 02:19:45.778894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:45.778861 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:46.254034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.254001 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" Oct 30 02:19:46.254539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.254036 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7" event={"ID":"ec619a8d-98ea-443c-a8b1-3ea4628805f3","Type":"ContainerDied","Data":"6b4f178a8eaf074fbf9f2e4db202711b8ea694de3d2c14e50a7ce79543b5bd0e"} Oct 30 02:19:46.254539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.254133 5998 scope.go:122] "RemoveContainer" containerID="49c33a7403ca1c38749bbd22f3950824153112de9948921beef267205e421738" Oct 30 02:19:46.323602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.323501 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7"] Oct 30 02:19:46.337612 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.337568 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-000edd/payload-processing-external-model-e2e-extmod-b-000edd-5ff8xcft7"] Oct 30 02:19:46.444579 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.444543 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 02:19:46.444825 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.444799 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" podUID="7076db80-f9a3-423f-b198-f0a7aba80bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 02:19:46.531590 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.531526 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx" Oct 30 02:19:46.535208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.535185 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh" Oct 30 02:19:46.540329 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.540303 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c156f00-37ed-4828-a0b6-19c3c985dfd2" path="/var/lib/kubelet/pods/6c156f00-37ed-4828-a0b6-19c3c985dfd2/volumes" Oct 30 02:19:46.540806 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.540790 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec619a8d-98ea-443c-a8b1-3ea4628805f3" path="/var/lib/kubelet/pods/ec619a8d-98ea-443c-a8b1-3ea4628805f3/volumes" Oct 30 02:19:46.645615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.645576 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-maas-api-openshift-ca\") pod \"1b1225a9-cd64-4a55-9051-a7fa3a16b3e7\" (UID: \"1b1225a9-cd64-4a55-9051-a7fa3a16b3e7\") " Oct 30 02:19:46.645813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.645759 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-maas-api-xks-ca\") pod \"1b1225a9-cd64-4a55-9051-a7fa3a16b3e7\" (UID: \"1b1225a9-cd64-4a55-9051-a7fa3a16b3e7\") " Oct 30 02:19:46.645893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.645814 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230d086e-bfe5-4c4a-805b-404cd9c5cba7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230d086e-bfe5-4c4a-805b-404cd9c5cba7-maas-api-openshift-ca\") pod \"230d086e-bfe5-4c4a-805b-404cd9c5cba7\" (UID: \"230d086e-bfe5-4c4a-805b-404cd9c5cba7\") " Oct 30 02:19:46.645947 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.645934 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-kube-api-access-kgll7\" (UniqueName: \"kubernetes.io/projected/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-kube-api-access-kgll7\") pod \"1b1225a9-cd64-4a55-9051-a7fa3a16b3e7\" (UID: \"1b1225a9-cd64-4a55-9051-a7fa3a16b3e7\") " Oct 30 02:19:46.646010 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.645965 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-maas-api-openshift-ca" pod "1b1225a9-cd64-4a55-9051-a7fa3a16b3e7" (UID: "1b1225a9-cd64-4a55-9051-a7fa3a16b3e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:46.646010 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.646002 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/230d086e-bfe5-4c4a-805b-404cd9c5cba7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230d086e-bfe5-4c4a-805b-404cd9c5cba7-maas-api-xks-ca\") pod \"230d086e-bfe5-4c4a-805b-404cd9c5cba7\" (UID: \"230d086e-bfe5-4c4a-805b-404cd9c5cba7\") " Oct 30 02:19:46.646134 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.646046 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230d086e-bfe5-4c4a-805b-404cd9c5cba7-kube-api-access-5qljg\" (UniqueName: \"kubernetes.io/projected/230d086e-bfe5-4c4a-805b-404cd9c5cba7-kube-api-access-5qljg\") pod \"230d086e-bfe5-4c4a-805b-404cd9c5cba7\" (UID: \"230d086e-bfe5-4c4a-805b-404cd9c5cba7\") " Oct 30 02:19:46.646783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.646532 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230d086e-bfe5-4c4a-805b-404cd9c5cba7-maas-api-openshift-ca" pod "230d086e-bfe5-4c4a-805b-404cd9c5cba7" (UID: "230d086e-bfe5-4c4a-805b-404cd9c5cba7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:46.646783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.646722 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230d086e-bfe5-4c4a-805b-404cd9c5cba7-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:46.646783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.646743 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:46.648445 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.648417 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-kube-api-access-kgll7" pod "1b1225a9-cd64-4a55-9051-a7fa3a16b3e7" (UID: "1b1225a9-cd64-4a55-9051-a7fa3a16b3e7"). InnerVolumeSpecName "kube-api-access-kgll7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:46.648592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.648570 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-maas-api-xks-ca" pod "1b1225a9-cd64-4a55-9051-a7fa3a16b3e7" (UID: "1b1225a9-cd64-4a55-9051-a7fa3a16b3e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:46.648720 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.648668 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230d086e-bfe5-4c4a-805b-404cd9c5cba7-kube-api-access-5qljg" pod "230d086e-bfe5-4c4a-805b-404cd9c5cba7" (UID: "230d086e-bfe5-4c4a-805b-404cd9c5cba7"). InnerVolumeSpecName "kube-api-access-5qljg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:46.648922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.648906 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/230d086e-bfe5-4c4a-805b-404cd9c5cba7-maas-api-xks-ca" pod "230d086e-bfe5-4c4a-805b-404cd9c5cba7" (UID: "230d086e-bfe5-4c4a-805b-404cd9c5cba7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:46.748663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.748618 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgll7\" (UniqueName: \"kubernetes.io/projected/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-kube-api-access-kgll7\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:46.748867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.748670 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230d086e-bfe5-4c4a-805b-404cd9c5cba7-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:46.748867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.748688 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qljg\" (UniqueName: \"kubernetes.io/projected/230d086e-bfe5-4c4a-805b-404cd9c5cba7-kube-api-access-5qljg\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:46.748867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:46.748711 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:47.264428 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:47.264396 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh" Oct 30 02:19:47.264428 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:47.264420 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh" event={"ID":"1b1225a9-cd64-4a55-9051-a7fa3a16b3e7","Type":"ContainerDied","Data":"453dfba678f8b663033d2dc536cf9acac127231b33ca68f1916b8258792bcaf3"} Oct 30 02:19:47.265003 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:47.264470 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="453dfba678f8b663033d2dc536cf9acac127231b33ca68f1916b8258792bcaf3" Oct 30 02:19:47.265953 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:47.265930 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx" Oct 30 02:19:47.266126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:47.265955 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx" event={"ID":"230d086e-bfe5-4c4a-805b-404cd9c5cba7","Type":"ContainerDied","Data":"565c03a525444c0e1192ac2b9be52c0bc9d60d5fc64a9ed8fe9c5553d6d42576"} Oct 30 02:19:47.266126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:47.266006 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="565c03a525444c0e1192ac2b9be52c0bc9d60d5fc64a9ed8fe9c5553d6d42576" Oct 30 02:19:47.379474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:47.379440 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:47.464035 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:47.463995 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:19:47.465576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:47.465551 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:19:48.957348 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:48.957311 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:49.884432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:49.884393 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:50.309116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.308512 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x"] Oct 30 02:19:50.324125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.322716 5998 cpu_manager.go:414] "containerMap: removing container" podUID="230d086e-bfe5-4c4a-805b-404cd9c5cba7" containerName="revoke-keys" Oct 30 02:19:50.324125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.322747 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230d086e-bfe5-4c4a-805b-404cd9c5cba7" containerName="revoke-keys" Oct 30 02:19:50.324125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.322773 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ec619a8d-98ea-443c-a8b1-3ea4628805f3" containerName="payload-processing" Oct 30 02:19:50.324125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.322782 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec619a8d-98ea-443c-a8b1-3ea4628805f3" containerName="payload-processing" Oct 30 02:19:50.324125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.322831 5998 cpu_manager.go:414] "containerMap: removing container" podUID="6c156f00-37ed-4828-a0b6-19c3c985dfd2" containerName="payload-processing" Oct 30 02:19:50.324125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.322839 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c156f00-37ed-4828-a0b6-19c3c985dfd2" containerName="payload-processing" Oct 30 02:19:50.324125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.322859 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1b1225a9-cd64-4a55-9051-a7fa3a16b3e7" containerName="revoke-keys" Oct 30 02:19:50.324125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.322866 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b1225a9-cd64-4a55-9051-a7fa3a16b3e7" containerName="revoke-keys" Oct 30 02:19:50.324125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.323006 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c156f00-37ed-4828-a0b6-19c3c985dfd2" containerName="payload-processing" Oct 30 02:19:50.324125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.323021 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b1225a9-cd64-4a55-9051-a7fa3a16b3e7" containerName="revoke-keys" Oct 30 02:19:50.324125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.323038 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="230d086e-bfe5-4c4a-805b-404cd9c5cba7" containerName="revoke-keys" Oct 30 02:19:50.324125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.323049 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec619a8d-98ea-443c-a8b1-3ea4628805f3" containerName="payload-processing" Oct 30 02:19:50.355654 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.355622 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x"] Oct 30 02:19:50.356006 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.355943 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.375989 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.375960 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-9dfb3c-openshift-default-dockercfg-r8wg4\"" Oct 30 02:19:50.506572 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.506537 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:50.517609 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.517571 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-workload-certs\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.517940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.517914 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-workload-socket\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.518171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.518149 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-token\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.518370 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.518350 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-credential-socket\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.518604 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.518586 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b71a2d-53c4-4e41-85ff-b82b82292598-istiod-ca-cert\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.519208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.519076 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-data\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.519443 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.519424 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-podinfo\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.519663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.519642 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkmzt\" (UniqueName: \"kubernetes.io/projected/d1b71a2d-53c4-4e41-85ff-b82b82292598-kube-api-access-kkmzt\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.520049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.520027 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-envoy\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.621484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.621385 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-workload-certs\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.621484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.621441 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-workload-socket\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.621734 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.621579 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-token\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.621734 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.621634 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-credential-socket\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.621734 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.621724 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b71a2d-53c4-4e41-85ff-b82b82292598-istiod-ca-cert\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.621912 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.621834 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-data\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.621912 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.621862 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-podinfo\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.621912 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.621871 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-workload-socket\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.622262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.621906 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-workload-certs\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.622262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.621974 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkmzt\" (UniqueName: \"kubernetes.io/projected/d1b71a2d-53c4-4e41-85ff-b82b82292598-kube-api-access-kkmzt\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.622262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.622075 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-envoy\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.622262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.622198 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-credential-socket\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.622262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.622257 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-data\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.623223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.623191 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b71a2d-53c4-4e41-85ff-b82b82292598-istiod-ca-cert\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.625126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.625053 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-envoy\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.625433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.625406 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-podinfo\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.643044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.643012 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-token\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.643257 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.643237 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkmzt\" (UniqueName: \"kubernetes.io/projected/d1b71a2d-53c4-4e41-85ff-b82b82292598-kube-api-access-kkmzt\") pod \"e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:50.687350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:50.687311 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:51.012198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:51.012171 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x"] Oct 30 02:19:51.014929 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:19:51.014899 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b71a2d_53c4_4e41_85ff_b82b82292598.slice/crio-ff158feacde0ca130b920b2efeb0b40b183928d9129c6be22d316b6c34626a20 WatchSource:0}: Error finding container ff158feacde0ca130b920b2efeb0b40b183928d9129c6be22d316b6c34626a20: Status 404 returned error can't find the container with id ff158feacde0ca130b920b2efeb0b40b183928d9129c6be22d316b6c34626a20 Oct 30 02:19:51.018387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:51.018347 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:19:51.018481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:51.018449 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:19:51.018533 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:51.018492 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:19:51.336423 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:51.336315 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" event={"ID":"d1b71a2d-53c4-4e41-85ff-b82b82292598","Type":"ContainerStarted","Data":"958467014e0618b5aeab19dc44c3175db9f4679fd947bc3af0f2b569ee3c2842"} Oct 30 02:19:51.336423 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:51.336388 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" event={"ID":"d1b71a2d-53c4-4e41-85ff-b82b82292598","Type":"ContainerStarted","Data":"ff158feacde0ca130b920b2efeb0b40b183928d9129c6be22d316b6c34626a20"} Oct 30 02:19:51.413440 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:51.409938 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" podStartSLOduration=1.409913522 podStartE2EDuration="1.409913522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:19:51.403943811 +0000 UTC m=+3020.982734893" watchObservedRunningTime="2026-10-30 02:19:51.409913522 +0000 UTC m=+3020.988704604" Oct 30 02:19:51.688071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:51.688037 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:52.352111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:52.351691 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:52.640357 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:52.640266 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:52.690240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:52.690197 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 02:19:52.690438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:52.690266 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" podUID="d1b71a2d-53c4-4e41-85ff-b82b82292598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 02:19:52.755358 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:52.755313 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:53.688322 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:53.688284 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 02:19:53.688738 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:53.688338 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" podUID="d1b71a2d-53c4-4e41-85ff-b82b82292598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 02:19:53.885414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:53.885373 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:54.019205 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.007742 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:54.127110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.126557 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:54.335774 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.335673 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr"] Oct 30 02:19:54.339369 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.339344 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" Oct 30 02:19:54.350676 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.350642 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:54.365851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.365814 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2"] Oct 30 02:19:54.369079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.369061 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" Oct 30 02:19:54.391521 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.391493 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2"] Oct 30 02:19:54.399218 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.399188 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d48d165-8ac6-4450-b354-1edfe69adbad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2\" (UID: \"4d48d165-8ac6-4450-b354-1edfe69adbad\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" Oct 30 02:19:54.399365 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.399230 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frf4j\" (UniqueName: \"kubernetes.io/projected/4d48d165-8ac6-4450-b354-1edfe69adbad-kube-api-access-frf4j\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2\" (UID: \"4d48d165-8ac6-4450-b354-1edfe69adbad\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" Oct 30 02:19:54.399365 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.399276 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d48d165-8ac6-4450-b354-1edfe69adbad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2\" (UID: \"4d48d165-8ac6-4450-b354-1edfe69adbad\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" Oct 30 02:19:54.399365 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.399325 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d48d165-8ac6-4450-b354-1edfe69adbad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2\" (UID: \"4d48d165-8ac6-4450-b354-1edfe69adbad\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" Oct 30 02:19:54.524113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.500574 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d48d165-8ac6-4450-b354-1edfe69adbad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2\" (UID: \"4d48d165-8ac6-4450-b354-1edfe69adbad\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" Oct 30 02:19:54.524113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.500632 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frf4j\" (UniqueName: \"kubernetes.io/projected/4d48d165-8ac6-4450-b354-1edfe69adbad-kube-api-access-frf4j\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2\" (UID: \"4d48d165-8ac6-4450-b354-1edfe69adbad\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" Oct 30 02:19:54.524113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.500683 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d48d165-8ac6-4450-b354-1edfe69adbad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2\" (UID: \"4d48d165-8ac6-4450-b354-1edfe69adbad\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" Oct 30 02:19:54.524113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.500740 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d48d165-8ac6-4450-b354-1edfe69adbad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2\" (UID: \"4d48d165-8ac6-4450-b354-1edfe69adbad\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" Oct 30 02:19:54.524113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.510386 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d48d165-8ac6-4450-b354-1edfe69adbad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2\" (UID: \"4d48d165-8ac6-4450-b354-1edfe69adbad\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" Oct 30 02:19:54.524113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.511013 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d48d165-8ac6-4450-b354-1edfe69adbad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2\" (UID: \"4d48d165-8ac6-4450-b354-1edfe69adbad\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" Oct 30 02:19:54.541728 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.541690 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d48d165-8ac6-4450-b354-1edfe69adbad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2\" (UID: \"4d48d165-8ac6-4450-b354-1edfe69adbad\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" Oct 30 02:19:54.585287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.576383 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp"] Oct 30 02:19:54.586551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.586393 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frf4j\" (UniqueName: \"kubernetes.io/projected/4d48d165-8ac6-4450-b354-1edfe69adbad-kube-api-access-frf4j\") pod \"payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2\" (UID: \"4d48d165-8ac6-4450-b354-1edfe69adbad\") " pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" Oct 30 02:19:54.588381 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.588361 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp" Oct 30 02:19:54.642114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.612424 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:19:54.642114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.613082 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:19:54.642114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.613212 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:19:54.642114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.613305 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:19:54.642114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.614599 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-006df12a-ghvtp\" (UID: \"08ec49c9-f31e-4e0f-ab12-599eb9121f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp" Oct 30 02:19:54.642114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.614695 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-006df12a-ghvtp\" (UID: \"08ec49c9-f31e-4e0f-ab12-599eb9121f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp" Oct 30 02:19:54.642114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.614829 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6w6v\" (UniqueName: \"kubernetes.io/projected/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-kube-api-access-p6w6v\") pod \"maas-api-revoke-sub-006df12a-ghvtp\" (UID: \"08ec49c9-f31e-4e0f-ab12-599eb9121f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp" Oct 30 02:19:54.642114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.628785 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp"] Oct 30 02:19:54.692818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.692787 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" Oct 30 02:19:54.780711 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.719200 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-006df12a-ghvtp\" (UID: \"08ec49c9-f31e-4e0f-ab12-599eb9121f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp" Oct 30 02:19:54.850493 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.850374 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-006df12a-ghvtp\" (UID: \"08ec49c9-f31e-4e0f-ab12-599eb9121f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp" Oct 30 02:19:54.850967 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.850918 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6w6v\" (UniqueName: \"kubernetes.io/projected/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-kube-api-access-p6w6v\") pod \"maas-api-revoke-sub-006df12a-ghvtp\" (UID: \"08ec49c9-f31e-4e0f-ab12-599eb9121f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp" Oct 30 02:19:54.878131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.852376 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-006df12a-ghvtp\" (UID: \"08ec49c9-f31e-4e0f-ab12-599eb9121f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp" Oct 30 02:19:54.878131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.750689 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-006df12a-ghvtp\" (UID: \"08ec49c9-f31e-4e0f-ab12-599eb9121f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp" Oct 30 02:19:54.878131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.768017 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 02:19:54.878131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.852459 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" podUID="d1b71a2d-53c4-4e41-85ff-b82b82292598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 02:19:54.878131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.799070 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:54.994747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.994716 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6w6v\" (UniqueName: \"kubernetes.io/projected/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-kube-api-access-p6w6v\") pod \"maas-api-revoke-sub-006df12a-ghvtp\" (UID: \"08ec49c9-f31e-4e0f-ab12-599eb9121f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp" Oct 30 02:19:54.999467 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:54.999438 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp" Oct 30 02:19:55.029032 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:55.028939 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:55.138799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:55.138678 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:55.263384 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:19:55.263346 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:19:55.341872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:55.340787 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:55.584413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:55.583955 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2"] Oct 30 02:19:55.642813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:55.642602 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp"] Oct 30 02:19:55.645156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:55.645076 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:55.647012 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:19:55.646972 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ec49c9_f31e_4e0f_ab12_599eb9121f8c.slice/crio-b7487278465bda1110ed61375e1f9785ee2ed1567a14de28ec9833cb3ee5dd27 WatchSource:0}: Error finding container b7487278465bda1110ed61375e1f9785ee2ed1567a14de28ec9833cb3ee5dd27: Status 404 returned error can't find the container with id b7487278465bda1110ed61375e1f9785ee2ed1567a14de28ec9833cb3ee5dd27 Oct 30 02:19:55.688859 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:55.688821 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 02:19:55.689011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:55.688871 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" podUID="d1b71a2d-53c4-4e41-85ff-b82b82292598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 02:19:55.985064 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:55.985019 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:56.026040 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.026000 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg"] Oct 30 02:19:56.048421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.047420 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.071192 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.068779 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-2c4b44-openshift-default-dockercfg-gqcdt\"" Oct 30 02:19:56.083980 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.083943 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg"] Oct 30 02:19:56.208129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.202938 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:56.208129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.204894 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b13ac864-0406-4db5-b911-2851f86df2b6-istio-podinfo\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.208129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.204935 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhvp8\" (UniqueName: \"kubernetes.io/projected/b13ac864-0406-4db5-b911-2851f86df2b6-kube-api-access-dhvp8\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.208129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.204999 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b13ac864-0406-4db5-b911-2851f86df2b6-istiod-ca-cert\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.208129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.205027 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-credential-socket\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.208129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.205075 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b13ac864-0406-4db5-b911-2851f86df2b6-istio-token\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.208129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.205133 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-workload-certs\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.208129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.205181 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-istio-data\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.208129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.205284 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-workload-socket\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.208129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.205327 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-istio-envoy\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.309040 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.308978 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-credential-socket\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.309338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.309316 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b13ac864-0406-4db5-b911-2851f86df2b6-istio-token\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.309945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.309922 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-workload-certs\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.310110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.310080 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-istio-data\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.310321 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.310307 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-workload-socket\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.310444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.310423 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-istio-envoy\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.310677 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.310663 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b13ac864-0406-4db5-b911-2851f86df2b6-istio-podinfo\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.310777 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.310763 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhvp8\" (UniqueName: \"kubernetes.io/projected/b13ac864-0406-4db5-b911-2851f86df2b6-kube-api-access-dhvp8\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.310883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.310871 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b13ac864-0406-4db5-b911-2851f86df2b6-istiod-ca-cert\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.311793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.311767 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b13ac864-0406-4db5-b911-2851f86df2b6-istiod-ca-cert\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.312591 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.312570 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-credential-socket\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.313250 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.313231 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-workload-certs\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.313584 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.313568 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-istio-data\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.313902 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.313887 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-workload-socket\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.327393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.325902 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-istio-envoy\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.411221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.404379 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:56.425110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.415035 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b13ac864-0406-4db5-b911-2851f86df2b6-istio-token\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.425110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.423886 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b13ac864-0406-4db5-b911-2851f86df2b6-istio-podinfo\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.446246 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.446207 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhvp8\" (UniqueName: \"kubernetes.io/projected/b13ac864-0406-4db5-b911-2851f86df2b6-kube-api-access-dhvp8\") pod \"e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:56.499386 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.499279 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" event={"ID":"4d48d165-8ac6-4450-b354-1edfe69adbad","Type":"ContainerStarted","Data":"7bc41a497b581957b02b325c5854d6ecbe784077997b90f3cef9e960c8272a25"} Oct 30 02:19:56.499712 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.499686 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" event={"ID":"4d48d165-8ac6-4450-b354-1edfe69adbad","Type":"ContainerStarted","Data":"0a972ea4fa82ddbf1e8ebd4d536b34a6349678fe29faf90df3dd7480058fb45c"} Oct 30 02:19:56.501952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.501732 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" Oct 30 02:19:56.512313 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.511515 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp" event={"ID":"08ec49c9-f31e-4e0f-ab12-599eb9121f8c","Type":"ContainerStarted","Data":"6b4a9d0e7e8d93386f929d99d6e5fd857e8e112c4a4864753552986467b64668"} Oct 30 02:19:56.512313 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.511573 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp" event={"ID":"08ec49c9-f31e-4e0f-ab12-599eb9121f8c","Type":"ContainerStarted","Data":"b7487278465bda1110ed61375e1f9785ee2ed1567a14de28ec9833cb3ee5dd27"} Oct 30 02:19:56.562469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.562400 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" podStartSLOduration=2.5623791430000002 podStartE2EDuration="2.562379143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:19:56.548515996 +0000 UTC m=+3026.127307081" watchObservedRunningTime="2026-10-30 02:19:56.562379143 +0000 UTC m=+3026.141170225" Oct 30 02:19:56.585914 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.585865 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp" podStartSLOduration=2.585851686 podStartE2EDuration="2.585851686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:19:56.585073157 +0000 UTC m=+3026.163864237" watchObservedRunningTime="2026-10-30 02:19:56.585851686 +0000 UTC m=+3026.164642768" Oct 30 02:19:56.687962 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.687918 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 02:19:56.688149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.687975 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" podUID="d1b71a2d-53c4-4e41-85ff-b82b82292598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 02:19:56.708117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:56.707830 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:57.076296 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:57.076264 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg"] Oct 30 02:19:57.077714 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:19:57.077673 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb13ac864_0406_4db5_b911_2851f86df2b6.slice/crio-b006f99212ab45a0e8ceaeb34bcf93eee2fc055fc412385f0ae984873151dea0 WatchSource:0}: Error finding container b006f99212ab45a0e8ceaeb34bcf93eee2fc055fc412385f0ae984873151dea0: Status 404 returned error can't find the container with id b006f99212ab45a0e8ceaeb34bcf93eee2fc055fc412385f0ae984873151dea0 Oct 30 02:19:57.080529 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:57.080493 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:19:57.080632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:57.080562 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:19:57.080632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:57.080592 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:19:57.552079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:57.552039 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:57.557460 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:57.557418 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" event={"ID":"b13ac864-0406-4db5-b911-2851f86df2b6","Type":"ContainerStarted","Data":"81cb1590c843dc22421c710b134b517b7ca49efb367815ff446e77a825311f55"} Oct 30 02:19:57.557647 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:57.557481 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" event={"ID":"b13ac864-0406-4db5-b911-2851f86df2b6","Type":"ContainerStarted","Data":"b006f99212ab45a0e8ceaeb34bcf93eee2fc055fc412385f0ae984873151dea0"} Oct 30 02:19:57.563113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:57.561680 5998 generic.go:356] "Generic (PLEG): container finished" podID="08ec49c9-f31e-4e0f-ab12-599eb9121f8c" containerID="6b4a9d0e7e8d93386f929d99d6e5fd857e8e112c4a4864753552986467b64668" exitCode=0 Oct 30 02:19:57.563113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:57.562468 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp" event={"ID":"08ec49c9-f31e-4e0f-ab12-599eb9121f8c","Type":"ContainerDied","Data":"6b4a9d0e7e8d93386f929d99d6e5fd857e8e112c4a4864753552986467b64668"} Oct 30 02:19:57.633113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:57.631181 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" podStartSLOduration=2.6311603679999997 podStartE2EDuration="2.631160368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:19:57.628066942 +0000 UTC m=+3027.206858026" watchObservedRunningTime="2026-10-30 02:19:57.631160368 +0000 UTC m=+3027.209951451" Oct 30 02:19:57.688216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:57.688178 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 02:19:57.688418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:57.688240 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" podUID="d1b71a2d-53c4-4e41-85ff-b82b82292598" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 02:19:57.708411 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:57.708377 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:58.693290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:58.693249 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:58.693981 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:58.693954 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:58.694589 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:58.694569 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:19:58.715922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:58.715891 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:58.759058 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:58.759031 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp" Oct 30 02:19:58.783561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:58.783529 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-kube-api-access-p6w6v\" (UniqueName: \"kubernetes.io/projected/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-kube-api-access-p6w6v\") pod \"08ec49c9-f31e-4e0f-ab12-599eb9121f8c\" (UID: \"08ec49c9-f31e-4e0f-ab12-599eb9121f8c\") " Oct 30 02:19:58.783737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:58.783569 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-maas-api-openshift-ca\") pod \"08ec49c9-f31e-4e0f-ab12-599eb9121f8c\" (UID: \"08ec49c9-f31e-4e0f-ab12-599eb9121f8c\") " Oct 30 02:19:58.783737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:58.783670 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-maas-api-xks-ca\") pod \"08ec49c9-f31e-4e0f-ab12-599eb9121f8c\" (UID: \"08ec49c9-f31e-4e0f-ab12-599eb9121f8c\") " Oct 30 02:19:58.784045 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:58.784017 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-maas-api-openshift-ca" pod "08ec49c9-f31e-4e0f-ab12-599eb9121f8c" (UID: "08ec49c9-f31e-4e0f-ab12-599eb9121f8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:58.784490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:58.784421 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:58.785780 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:58.785758 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-kube-api-access-p6w6v" pod "08ec49c9-f31e-4e0f-ab12-599eb9121f8c" (UID: "08ec49c9-f31e-4e0f-ab12-599eb9121f8c"). InnerVolumeSpecName "kube-api-access-p6w6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:58.786223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:58.786202 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-maas-api-xks-ca" pod "08ec49c9-f31e-4e0f-ab12-599eb9121f8c" (UID: "08ec49c9-f31e-4e0f-ab12-599eb9121f8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:58.886495 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:58.886401 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6w6v\" (UniqueName: \"kubernetes.io/projected/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-kube-api-access-p6w6v\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:58.886495 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:58.886442 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ec49c9-f31e-4e0f-ab12-599eb9121f8c-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.031779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.031737 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4"] Oct 30 02:19:59.035698 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.035662 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4877029e-j8vk4"] Oct 30 02:19:59.278442 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.278402 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:19:59.338210 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.338162 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" podUID="dc2f4f37-9a00-420f-9020-d6c74b917861" containerName="payload-processing" containerID="cri-o://605c4701997e3ef71474bfea3849ed211f43657cef2e8274df1fdf6754b99512" gracePeriod=25 Oct 30 02:19:59.581525 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.581488 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp" Oct 30 02:19:59.581525 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.581504 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp" event={"ID":"08ec49c9-f31e-4e0f-ab12-599eb9121f8c","Type":"ContainerDied","Data":"b7487278465bda1110ed61375e1f9785ee2ed1567a14de28ec9833cb3ee5dd27"} Oct 30 02:19:59.581806 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.581553 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7487278465bda1110ed61375e1f9785ee2ed1567a14de28ec9833cb3ee5dd27" Oct 30 02:19:59.583759 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.583730 5998 generic.go:356] "Generic (PLEG): container finished" podID="dc2f4f37-9a00-420f-9020-d6c74b917861" containerID="605c4701997e3ef71474bfea3849ed211f43657cef2e8274df1fdf6754b99512" exitCode=0 Oct 30 02:19:59.584193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.584162 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" event={"ID":"dc2f4f37-9a00-420f-9020-d6c74b917861","Type":"ContainerDied","Data":"605c4701997e3ef71474bfea3849ed211f43657cef2e8274df1fdf6754b99512"} Oct 30 02:19:59.584503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.584483 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:59.588351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.588330 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:19:59.610004 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.609979 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" Oct 30 02:19:59.697170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.697130 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc2f4f37-9a00-420f-9020-d6c74b917861-kube-api-access-scxm8\" (UniqueName: \"kubernetes.io/projected/dc2f4f37-9a00-420f-9020-d6c74b917861-kube-api-access-scxm8\") pod \"dc2f4f37-9a00-420f-9020-d6c74b917861\" (UID: \"dc2f4f37-9a00-420f-9020-d6c74b917861\") " Oct 30 02:19:59.697655 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.697223 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc2f4f37-9a00-420f-9020-d6c74b917861-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc2f4f37-9a00-420f-9020-d6c74b917861-provider-credentials\") pod \"dc2f4f37-9a00-420f-9020-d6c74b917861\" (UID: \"dc2f4f37-9a00-420f-9020-d6c74b917861\") " Oct 30 02:19:59.697655 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.697309 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc2f4f37-9a00-420f-9020-d6c74b917861-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc2f4f37-9a00-420f-9020-d6c74b917861-plugins-config-volume\") pod \"dc2f4f37-9a00-420f-9020-d6c74b917861\" (UID: \"dc2f4f37-9a00-420f-9020-d6c74b917861\") " Oct 30 02:19:59.697655 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.697406 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc2f4f37-9a00-420f-9020-d6c74b917861-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc2f4f37-9a00-420f-9020-d6c74b917861-routing-overlay-volume\") pod \"dc2f4f37-9a00-420f-9020-d6c74b917861\" (UID: \"dc2f4f37-9a00-420f-9020-d6c74b917861\") " Oct 30 02:19:59.697842 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.697744 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc2f4f37-9a00-420f-9020-d6c74b917861-plugins-config-volume" pod "dc2f4f37-9a00-420f-9020-d6c74b917861" (UID: "dc2f4f37-9a00-420f-9020-d6c74b917861"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:59.697976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.697947 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc2f4f37-9a00-420f-9020-d6c74b917861-routing-overlay-volume" pod "dc2f4f37-9a00-420f-9020-d6c74b917861" (UID: "dc2f4f37-9a00-420f-9020-d6c74b917861"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:59.699608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.699584 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2f4f37-9a00-420f-9020-d6c74b917861-provider-credentials" pod "dc2f4f37-9a00-420f-9020-d6c74b917861" (UID: "dc2f4f37-9a00-420f-9020-d6c74b917861"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:59.702020 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.700638 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2f4f37-9a00-420f-9020-d6c74b917861-kube-api-access-scxm8" pod "dc2f4f37-9a00-420f-9020-d6c74b917861" (UID: "dc2f4f37-9a00-420f-9020-d6c74b917861"). InnerVolumeSpecName "kube-api-access-scxm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:59.703621 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.703586 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scxm8\" (UniqueName: \"kubernetes.io/projected/dc2f4f37-9a00-420f-9020-d6c74b917861-kube-api-access-scxm8\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.703621 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.703616 5998 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc2f4f37-9a00-420f-9020-d6c74b917861-provider-credentials\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.703793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.703630 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc2f4f37-9a00-420f-9020-d6c74b917861-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.703793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:19:59.703643 5998 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc2f4f37-9a00-420f-9020-d6c74b917861-routing-overlay-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:20:00.022001 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:00.021969 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:00.140056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:00.140017 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:00.542308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:00.542271 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc72bfa4-a197-49da-803b-1336391e40f8" path="/var/lib/kubelet/pods/fc72bfa4-a197-49da-803b-1336391e40f8/volumes" Oct 30 02:20:00.592370 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:00.592338 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" Oct 30 02:20:00.592370 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:00.592361 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr" event={"ID":"dc2f4f37-9a00-420f-9020-d6c74b917861","Type":"ContainerDied","Data":"590b6e6a58a4b3c55499db2c8cc99e9e93b0834e12d89ea7462a2b9dba3cfc4d"} Oct 30 02:20:00.592596 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:00.592418 5998 scope.go:122] "RemoveContainer" containerID="605c4701997e3ef71474bfea3849ed211f43657cef2e8274df1fdf6754b99512" Oct 30 02:20:00.661430 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:00.661396 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr"] Oct 30 02:20:00.670809 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:00.670780 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-5cd4l4hhr"] Oct 30 02:20:01.098673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:01.098641 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:01.683558 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:01.683525 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:01.798048 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:01.798018 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:02.540590 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:02.540550 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc2f4f37-9a00-420f-9020-d6c74b917861" path="/var/lib/kubelet/pods/dc2f4f37-9a00-420f-9020-d6c74b917861/volumes" Oct 30 02:20:02.659285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:02.659244 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:02.788325 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:02.788291 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:02.896775 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:02.896679 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:04.585214 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:04.585180 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:06.055629 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:06.055590 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:06.100772 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:06.100733 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2"] Oct 30 02:20:06.104844 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:06.104794 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" Oct 30 02:20:06.168810 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:06.168776 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:06.290500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:06.290468 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:06.410641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:06.410615 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:06.705189 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:06.704567 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:08.147616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:08.147578 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:09.681990 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:09.681953 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:10.034782 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:10.034703 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7"] Oct 30 02:20:10.044818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:10.044787 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0114773b-jg4s7"] Oct 30 02:20:10.391596 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:10.391516 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:10.517489 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:10.517454 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:10.543603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:10.543568 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89f01e25-ada7-489b-966d-823f8bc623a1" path="/var/lib/kubelet/pods/89f01e25-ada7-489b-966d-823f8bc623a1/volumes" Oct 30 02:20:11.101774 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.101727 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" podUID="4d48d165-8ac6-4450-b354-1edfe69adbad" containerName="payload-processing" containerID="cri-o://7bc41a497b581957b02b325c5854d6ecbe784077997b90f3cef9e960c8272a25" gracePeriod=25 Oct 30 02:20:11.280489 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.280426 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:11.403516 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.403451 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:11.437488 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.437457 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" Oct 30 02:20:11.450700 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.450663 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d48d165-8ac6-4450-b354-1edfe69adbad-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d48d165-8ac6-4450-b354-1edfe69adbad-routing-overlay-volume\") pod \"4d48d165-8ac6-4450-b354-1edfe69adbad\" (UID: \"4d48d165-8ac6-4450-b354-1edfe69adbad\") " Oct 30 02:20:11.451371 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.451241 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d48d165-8ac6-4450-b354-1edfe69adbad-routing-overlay-volume" pod "4d48d165-8ac6-4450-b354-1edfe69adbad" (UID: "4d48d165-8ac6-4450-b354-1edfe69adbad"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:11.451718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.451685 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d48d165-8ac6-4450-b354-1edfe69adbad-kube-api-access-frf4j\" (UniqueName: \"kubernetes.io/projected/4d48d165-8ac6-4450-b354-1edfe69adbad-kube-api-access-frf4j\") pod \"4d48d165-8ac6-4450-b354-1edfe69adbad\" (UID: \"4d48d165-8ac6-4450-b354-1edfe69adbad\") " Oct 30 02:20:11.451839 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.451767 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d48d165-8ac6-4450-b354-1edfe69adbad-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d48d165-8ac6-4450-b354-1edfe69adbad-provider-credentials\") pod \"4d48d165-8ac6-4450-b354-1edfe69adbad\" (UID: \"4d48d165-8ac6-4450-b354-1edfe69adbad\") " Oct 30 02:20:11.451839 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.451808 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d48d165-8ac6-4450-b354-1edfe69adbad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d48d165-8ac6-4450-b354-1edfe69adbad-plugins-config-volume\") pod \"4d48d165-8ac6-4450-b354-1edfe69adbad\" (UID: \"4d48d165-8ac6-4450-b354-1edfe69adbad\") " Oct 30 02:20:11.452474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.452451 5998 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d48d165-8ac6-4450-b354-1edfe69adbad-routing-overlay-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.452616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.452595 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d48d165-8ac6-4450-b354-1edfe69adbad-plugins-config-volume" pod "4d48d165-8ac6-4450-b354-1edfe69adbad" (UID: "4d48d165-8ac6-4450-b354-1edfe69adbad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:11.454237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.454208 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d48d165-8ac6-4450-b354-1edfe69adbad-provider-credentials" pod "4d48d165-8ac6-4450-b354-1edfe69adbad" (UID: "4d48d165-8ac6-4450-b354-1edfe69adbad"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:11.478616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.478575 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d48d165-8ac6-4450-b354-1edfe69adbad-kube-api-access-frf4j" pod "4d48d165-8ac6-4450-b354-1edfe69adbad" (UID: "4d48d165-8ac6-4450-b354-1edfe69adbad"). InnerVolumeSpecName "kube-api-access-frf4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:11.553693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.553651 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frf4j\" (UniqueName: \"kubernetes.io/projected/4d48d165-8ac6-4450-b354-1edfe69adbad-kube-api-access-frf4j\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.553693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.553687 5998 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d48d165-8ac6-4450-b354-1edfe69adbad-provider-credentials\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.553693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.553699 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d48d165-8ac6-4450-b354-1edfe69adbad-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:20:11.674172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.660941 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:11.760746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.760711 5998 generic.go:356] "Generic (PLEG): container finished" podID="4d48d165-8ac6-4450-b354-1edfe69adbad" containerID="7bc41a497b581957b02b325c5854d6ecbe784077997b90f3cef9e960c8272a25" exitCode=0 Oct 30 02:20:11.761166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.761143 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" Oct 30 02:20:11.769231 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.761383 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" event={"ID":"4d48d165-8ac6-4450-b354-1edfe69adbad","Type":"ContainerDied","Data":"7bc41a497b581957b02b325c5854d6ecbe784077997b90f3cef9e960c8272a25"} Oct 30 02:20:11.769453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.769431 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2" event={"ID":"4d48d165-8ac6-4450-b354-1edfe69adbad","Type":"ContainerDied","Data":"0a972ea4fa82ddbf1e8ebd4d536b34a6349678fe29faf90df3dd7480058fb45c"} Oct 30 02:20:11.769559 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.769548 5998 scope.go:122] "RemoveContainer" containerID="7bc41a497b581957b02b325c5854d6ecbe784077997b90f3cef9e960c8272a25" Oct 30 02:20:11.867359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.867323 5998 scope.go:122] "RemoveContainer" containerID="7bc41a497b581957b02b325c5854d6ecbe784077997b90f3cef9e960c8272a25" Oct 30 02:20:11.867660 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.867464 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2"] Oct 30 02:20:11.869041 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:20:11.869011 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc41a497b581957b02b325c5854d6ecbe784077997b90f3cef9e960c8272a25\": container with ID starting with 7bc41a497b581957b02b325c5854d6ecbe784077997b90f3cef9e960c8272a25 not found: ID does not exist" containerID="7bc41a497b581957b02b325c5854d6ecbe784077997b90f3cef9e960c8272a25" Oct 30 02:20:11.869225 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.869062 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc41a497b581957b02b325c5854d6ecbe784077997b90f3cef9e960c8272a25"} err="failed to get container status \"7bc41a497b581957b02b325c5854d6ecbe784077997b90f3cef9e960c8272a25\": rpc error: code = NotFound desc = could not find container \"7bc41a497b581957b02b325c5854d6ecbe784077997b90f3cef9e960c8272a25\": container with ID starting with 7bc41a497b581957b02b325c5854d6ecbe784077997b90f3cef9e960c8272a25 not found: ID does not exist" Oct 30 02:20:11.871998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:11.870895 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-9dfb3c/payload-processing-external-model-e2e-extmod-b-9dfb3c-7fccbfmw2"] Oct 30 02:20:12.540542 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:12.540501 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d48d165-8ac6-4450-b354-1edfe69adbad" path="/var/lib/kubelet/pods/4d48d165-8ac6-4450-b354-1edfe69adbad/volumes" Oct 30 02:20:12.714491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:12.714451 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:13.589016 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:13.588984 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:14.003625 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.003589 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx"] Oct 30 02:20:14.004311 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.004288 5998 cpu_manager.go:414] "containerMap: removing container" podUID="dc2f4f37-9a00-420f-9020-d6c74b917861" containerName="payload-processing" Oct 30 02:20:14.004311 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.004309 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc2f4f37-9a00-420f-9020-d6c74b917861" containerName="payload-processing" Oct 30 02:20:14.004510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.004324 5998 cpu_manager.go:414] "containerMap: removing container" podUID="08ec49c9-f31e-4e0f-ab12-599eb9121f8c" containerName="revoke-keys" Oct 30 02:20:14.004510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.004333 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ec49c9-f31e-4e0f-ab12-599eb9121f8c" containerName="revoke-keys" Oct 30 02:20:14.004510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.004369 5998 cpu_manager.go:414] "containerMap: removing container" podUID="4d48d165-8ac6-4450-b354-1edfe69adbad" containerName="payload-processing" Oct 30 02:20:14.004510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.004377 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d48d165-8ac6-4450-b354-1edfe69adbad" containerName="payload-processing" Oct 30 02:20:14.004824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.004520 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc2f4f37-9a00-420f-9020-d6c74b917861" containerName="payload-processing" Oct 30 02:20:14.004824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.004535 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ec49c9-f31e-4e0f-ab12-599eb9121f8c" containerName="revoke-keys" Oct 30 02:20:14.004824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.004551 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d48d165-8ac6-4450-b354-1edfe69adbad" containerName="payload-processing" Oct 30 02:20:14.009396 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.009374 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx" Oct 30 02:20:14.041584 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.041555 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx"] Oct 30 02:20:14.043308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.042351 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:20:14.043308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.042490 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:20:14.043308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.042531 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:20:14.043308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.042558 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:20:14.047058 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.046816 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:14.117084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.117034 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa32e97-4a60-4890-ae3f-0a66123b81b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8ba2fbb-s98dx\" (UID: \"9fa32e97-4a60-4890-ae3f-0a66123b81b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx" Oct 30 02:20:14.117714 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.117681 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwcjj\" (UniqueName: \"kubernetes.io/projected/9fa32e97-4a60-4890-ae3f-0a66123b81b9-kube-api-access-zwcjj\") pod \"maas-api-revoke-sub-e8ba2fbb-s98dx\" (UID: \"9fa32e97-4a60-4890-ae3f-0a66123b81b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx" Oct 30 02:20:14.117925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.117907 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa32e97-4a60-4890-ae3f-0a66123b81b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8ba2fbb-s98dx\" (UID: \"9fa32e97-4a60-4890-ae3f-0a66123b81b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx" Oct 30 02:20:14.173720 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.173680 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:14.219238 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.219198 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwcjj\" (UniqueName: \"kubernetes.io/projected/9fa32e97-4a60-4890-ae3f-0a66123b81b9-kube-api-access-zwcjj\") pod \"maas-api-revoke-sub-e8ba2fbb-s98dx\" (UID: \"9fa32e97-4a60-4890-ae3f-0a66123b81b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx" Oct 30 02:20:14.219438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.219250 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa32e97-4a60-4890-ae3f-0a66123b81b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8ba2fbb-s98dx\" (UID: \"9fa32e97-4a60-4890-ae3f-0a66123b81b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx" Oct 30 02:20:14.219438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.219324 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa32e97-4a60-4890-ae3f-0a66123b81b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8ba2fbb-s98dx\" (UID: \"9fa32e97-4a60-4890-ae3f-0a66123b81b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx" Oct 30 02:20:14.220019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.219992 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa32e97-4a60-4890-ae3f-0a66123b81b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8ba2fbb-s98dx\" (UID: \"9fa32e97-4a60-4890-ae3f-0a66123b81b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx" Oct 30 02:20:14.221756 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.221734 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa32e97-4a60-4890-ae3f-0a66123b81b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8ba2fbb-s98dx\" (UID: \"9fa32e97-4a60-4890-ae3f-0a66123b81b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx" Oct 30 02:20:14.254239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.254174 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwcjj\" (UniqueName: \"kubernetes.io/projected/9fa32e97-4a60-4890-ae3f-0a66123b81b9-kube-api-access-zwcjj\") pod \"maas-api-revoke-sub-e8ba2fbb-s98dx\" (UID: \"9fa32e97-4a60-4890-ae3f-0a66123b81b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx" Oct 30 02:20:14.315937 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.315904 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:14.324588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.324557 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx" Oct 30 02:20:14.641460 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.641428 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx"] Oct 30 02:20:14.643436 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:20:14.643396 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa32e97_4a60_4890_ae3f_0a66123b81b9.slice/crio-5aaf1ae5b529a0a87744d8838b6456f3f7acfc387d2e2a9ab2d203163f53c1e7 WatchSource:0}: Error finding container 5aaf1ae5b529a0a87744d8838b6456f3f7acfc387d2e2a9ab2d203163f53c1e7: Status 404 returned error can't find the container with id 5aaf1ae5b529a0a87744d8838b6456f3f7acfc387d2e2a9ab2d203163f53c1e7 Oct 30 02:20:14.788917 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:14.788878 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx" event={"ID":"9fa32e97-4a60-4890-ae3f-0a66123b81b9","Type":"ContainerStarted","Data":"5aaf1ae5b529a0a87744d8838b6456f3f7acfc387d2e2a9ab2d203163f53c1e7"} Oct 30 02:20:15.188475 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:15.188394 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:15.798672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:15.798631 5998 generic.go:356] "Generic (PLEG): container finished" podID="9fa32e97-4a60-4890-ae3f-0a66123b81b9" containerID="936543a0b463010a691b88c9797ddfc3f8c274ac45fa7d3468bd18b5c6888cbf" exitCode=0 Oct 30 02:20:15.799063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:15.798716 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx" event={"ID":"9fa32e97-4a60-4890-ae3f-0a66123b81b9","Type":"ContainerDied","Data":"936543a0b463010a691b88c9797ddfc3f8c274ac45fa7d3468bd18b5c6888cbf"} Oct 30 02:20:16.756022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:16.755987 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:17.033327 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:17.033301 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx" Oct 30 02:20:17.066144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:17.066074 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fa32e97-4a60-4890-ae3f-0a66123b81b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa32e97-4a60-4890-ae3f-0a66123b81b9-maas-api-xks-ca\") pod \"9fa32e97-4a60-4890-ae3f-0a66123b81b9\" (UID: \"9fa32e97-4a60-4890-ae3f-0a66123b81b9\") " Oct 30 02:20:17.066308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:17.066290 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa32e97-4a60-4890-ae3f-0a66123b81b9-kube-api-access-zwcjj\" (UniqueName: \"kubernetes.io/projected/9fa32e97-4a60-4890-ae3f-0a66123b81b9-kube-api-access-zwcjj\") pod \"9fa32e97-4a60-4890-ae3f-0a66123b81b9\" (UID: \"9fa32e97-4a60-4890-ae3f-0a66123b81b9\") " Oct 30 02:20:17.066353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:17.066313 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa32e97-4a60-4890-ae3f-0a66123b81b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa32e97-4a60-4890-ae3f-0a66123b81b9-maas-api-openshift-ca\") pod \"9fa32e97-4a60-4890-ae3f-0a66123b81b9\" (UID: \"9fa32e97-4a60-4890-ae3f-0a66123b81b9\") " Oct 30 02:20:17.066778 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:17.066746 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa32e97-4a60-4890-ae3f-0a66123b81b9-maas-api-openshift-ca" pod "9fa32e97-4a60-4890-ae3f-0a66123b81b9" (UID: "9fa32e97-4a60-4890-ae3f-0a66123b81b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:17.068200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:17.068171 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fa32e97-4a60-4890-ae3f-0a66123b81b9-maas-api-xks-ca" pod "9fa32e97-4a60-4890-ae3f-0a66123b81b9" (UID: "9fa32e97-4a60-4890-ae3f-0a66123b81b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:20:17.068359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:17.068340 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa32e97-4a60-4890-ae3f-0a66123b81b9-kube-api-access-zwcjj" pod "9fa32e97-4a60-4890-ae3f-0a66123b81b9" (UID: "9fa32e97-4a60-4890-ae3f-0a66123b81b9"). InnerVolumeSpecName "kube-api-access-zwcjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:17.168339 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:17.168291 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwcjj\" (UniqueName: \"kubernetes.io/projected/9fa32e97-4a60-4890-ae3f-0a66123b81b9-kube-api-access-zwcjj\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:20:17.168339 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:17.168332 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fa32e97-4a60-4890-ae3f-0a66123b81b9-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:20:17.168611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:17.168349 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fa32e97-4a60-4890-ae3f-0a66123b81b9-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:20:17.858560 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:17.851181 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx" Oct 30 02:20:17.858560 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:17.851214 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx" event={"ID":"9fa32e97-4a60-4890-ae3f-0a66123b81b9","Type":"ContainerDied","Data":"5aaf1ae5b529a0a87744d8838b6456f3f7acfc387d2e2a9ab2d203163f53c1e7"} Oct 30 02:20:17.858560 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:17.851278 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aaf1ae5b529a0a87744d8838b6456f3f7acfc387d2e2a9ab2d203163f53c1e7" Oct 30 02:20:18.258063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:18.258027 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:19.806417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:19.806378 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:20.837525 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:20.837489 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:20.986118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:20.986061 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:21.120406 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:21.120324 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:21.289328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:21.289296 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:21.457430 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:21.457392 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:21.584008 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:21.583972 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:21.772260 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:21.772049 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:22.452275 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:22.451551 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:22.575218 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:22.575181 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:23.465011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:23.464976 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:23.715148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:23.715036 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:24.897705 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:24.897669 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:25.073704 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:25.073663 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:25.157070 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:25.156976 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:20:25.222506 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:20:25.222456 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:20:25.291499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:25.291466 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:25.409528 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:25.409406 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:25.986133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:25.983503 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g"] Oct 30 02:20:26.015066 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:26.015027 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:26.135275 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:26.135243 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:26.296801 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:26.296687 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:26.708394 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:26.708288 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:26.860217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:26.860183 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:26.990473 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:26.990375 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:27.163148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:27.163079 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:27.389665 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:27.389568 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:27.529934 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:27.529890 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:28.204815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:28.204772 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:28.323563 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:28.323525 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:29.432715 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:29.432618 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:29.571689 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:29.571643 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:29.706165 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:29.706054 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:30.031840 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.031761 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:30.309891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.309807 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn"] Oct 30 02:20:30.310452 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.310428 5998 cpu_manager.go:414] "containerMap: removing container" podUID="9fa32e97-4a60-4890-ae3f-0a66123b81b9" containerName="revoke-keys" Oct 30 02:20:30.310525 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.310453 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa32e97-4a60-4890-ae3f-0a66123b81b9" containerName="revoke-keys" Oct 30 02:20:30.310588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.310579 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa32e97-4a60-4890-ae3f-0a66123b81b9" containerName="revoke-keys" Oct 30 02:20:30.315157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.315136 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" Oct 30 02:20:30.338827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.338801 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"agc-cred-80a3dc502ca9\"" Oct 30 02:20:30.339069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.339047 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"payload-processing-external-model-e2e--1bd5e061-dockercfg-jz6pw\"" Oct 30 02:20:30.343418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.343392 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn"] Oct 30 02:20:30.380021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.379991 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwglz\" (UniqueName: \"kubernetes.io/projected/525eb671-a86f-4f82-9bc8-7b53d2801c91-kube-api-access-xwglz\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn\" (UID: \"525eb671-a86f-4f82-9bc8-7b53d2801c91\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" Oct 30 02:20:30.380240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.380028 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/525eb671-a86f-4f82-9bc8-7b53d2801c91-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn\" (UID: \"525eb671-a86f-4f82-9bc8-7b53d2801c91\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" Oct 30 02:20:30.380240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.380210 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/525eb671-a86f-4f82-9bc8-7b53d2801c91-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn\" (UID: \"525eb671-a86f-4f82-9bc8-7b53d2801c91\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" Oct 30 02:20:30.380379 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.380331 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/525eb671-a86f-4f82-9bc8-7b53d2801c91-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn\" (UID: \"525eb671-a86f-4f82-9bc8-7b53d2801c91\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" Oct 30 02:20:30.485070 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.485020 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwglz\" (UniqueName: \"kubernetes.io/projected/525eb671-a86f-4f82-9bc8-7b53d2801c91-kube-api-access-xwglz\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn\" (UID: \"525eb671-a86f-4f82-9bc8-7b53d2801c91\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" Oct 30 02:20:30.485740 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.485707 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/525eb671-a86f-4f82-9bc8-7b53d2801c91-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn\" (UID: \"525eb671-a86f-4f82-9bc8-7b53d2801c91\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" Oct 30 02:20:30.486041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.486021 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/525eb671-a86f-4f82-9bc8-7b53d2801c91-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn\" (UID: \"525eb671-a86f-4f82-9bc8-7b53d2801c91\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" Oct 30 02:20:30.486299 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.486275 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/525eb671-a86f-4f82-9bc8-7b53d2801c91-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn\" (UID: \"525eb671-a86f-4f82-9bc8-7b53d2801c91\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" Oct 30 02:20:30.486832 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.486803 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/525eb671-a86f-4f82-9bc8-7b53d2801c91-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn\" (UID: \"525eb671-a86f-4f82-9bc8-7b53d2801c91\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" Oct 30 02:20:30.487532 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.487509 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/525eb671-a86f-4f82-9bc8-7b53d2801c91-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn\" (UID: \"525eb671-a86f-4f82-9bc8-7b53d2801c91\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" Oct 30 02:20:30.521975 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.521939 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/525eb671-a86f-4f82-9bc8-7b53d2801c91-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn\" (UID: \"525eb671-a86f-4f82-9bc8-7b53d2801c91\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" Oct 30 02:20:30.525273 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.525243 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwglz\" (UniqueName: \"kubernetes.io/projected/525eb671-a86f-4f82-9bc8-7b53d2801c91-kube-api-access-xwglz\") pod \"payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn\" (UID: \"525eb671-a86f-4f82-9bc8-7b53d2801c91\") " pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" Oct 30 02:20:30.643841 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.643421 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a0fc73\"/\"payload-processing-external-model-e2e--1bd5e061-dockercfg-jz6pw\"" Oct 30 02:20:30.643841 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.643530 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" Oct 30 02:20:30.983868 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.983805 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" podUID="b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4" containerName="payload-processing" containerID="cri-o://471a41f1e3af2d2e018072cb11709b8f8fdf6a7983607e69ab0e7ba70e7ef177" gracePeriod=25 Oct 30 02:20:30.989450 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:30.989423 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn"] Oct 30 02:20:30.991605 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:20:30.991550 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525eb671_a86f_4f82_9bc8_7b53d2801c91.slice/crio-039a05d610daddc33e65bdbcd740a645ab0ab262252e77919cea83d7d59af687 WatchSource:0}: Error finding container 039a05d610daddc33e65bdbcd740a645ab0ab262252e77919cea83d7d59af687: Status 404 returned error can't find the container with id 039a05d610daddc33e65bdbcd740a645ab0ab262252e77919cea83d7d59af687 Oct 30 02:20:31.052929 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:31.052891 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" event={"ID":"525eb671-a86f-4f82-9bc8-7b53d2801c91","Type":"ContainerStarted","Data":"039a05d610daddc33e65bdbcd740a645ab0ab262252e77919cea83d7d59af687"} Oct 30 02:20:31.261194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:31.261162 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" Oct 30 02:20:31.301055 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:31.301018 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-routing-overlay-volume\") pod \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\" (UID: \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\") " Oct 30 02:20:31.301280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:31.301077 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-kube-api-access-8pn67\" (UniqueName: \"kubernetes.io/projected/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-kube-api-access-8pn67\") pod \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\" (UID: \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\") " Oct 30 02:20:31.301280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:31.301257 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-provider-credentials\") pod \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\" (UID: \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\") " Oct 30 02:20:31.301443 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:31.301352 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-plugins-config-volume\") pod \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\" (UID: \"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4\") " Oct 30 02:20:31.301536 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:31.301462 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-routing-overlay-volume" pod "b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4" (UID: "b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:31.302016 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:31.301974 5998 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-routing-overlay-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:20:31.302435 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:31.302405 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-plugins-config-volume" pod "b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4" (UID: "b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:31.305910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:31.305882 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-kube-api-access-8pn67" pod "b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4" (UID: "b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4"). InnerVolumeSpecName "kube-api-access-8pn67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:31.306490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:31.306464 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-provider-credentials" pod "b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4" (UID: "b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:31.391525 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:31.391488 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:31.403337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:31.403301 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:20:31.403337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:31.403330 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pn67\" (UniqueName: \"kubernetes.io/projected/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-kube-api-access-8pn67\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:20:31.403337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:31.403342 5998 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4-provider-credentials\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:20:32.066259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:32.066225 5998 generic.go:356] "Generic (PLEG): container finished" podID="b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4" containerID="471a41f1e3af2d2e018072cb11709b8f8fdf6a7983607e69ab0e7ba70e7ef177" exitCode=0 Oct 30 02:20:32.066773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:32.066280 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" Oct 30 02:20:32.066773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:32.066301 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" event={"ID":"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4","Type":"ContainerDied","Data":"471a41f1e3af2d2e018072cb11709b8f8fdf6a7983607e69ab0e7ba70e7ef177"} Oct 30 02:20:32.066773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:32.066368 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g" event={"ID":"b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4","Type":"ContainerDied","Data":"0cca2eaeb93f84c1694055b5921673b58a84235b5155f6381bb2dc9bc5ba0d6e"} Oct 30 02:20:32.066773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:32.066403 5998 scope.go:122] "RemoveContainer" containerID="471a41f1e3af2d2e018072cb11709b8f8fdf6a7983607e69ab0e7ba70e7ef177" Oct 30 02:20:32.068155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:32.068123 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" event={"ID":"525eb671-a86f-4f82-9bc8-7b53d2801c91","Type":"ContainerStarted","Data":"ac9303f0772b1c8a33217ad7d1c13616205564c9f0997488c6b0b0cc150146f4"} Oct 30 02:20:32.068304 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:32.068187 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" Oct 30 02:20:32.077846 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:32.077826 5998 scope.go:122] "RemoveContainer" containerID="471a41f1e3af2d2e018072cb11709b8f8fdf6a7983607e69ab0e7ba70e7ef177" Oct 30 02:20:32.078139 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:20:32.078119 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471a41f1e3af2d2e018072cb11709b8f8fdf6a7983607e69ab0e7ba70e7ef177\": container with ID starting with 471a41f1e3af2d2e018072cb11709b8f8fdf6a7983607e69ab0e7ba70e7ef177 not found: ID does not exist" containerID="471a41f1e3af2d2e018072cb11709b8f8fdf6a7983607e69ab0e7ba70e7ef177" Oct 30 02:20:32.078202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:32.078148 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471a41f1e3af2d2e018072cb11709b8f8fdf6a7983607e69ab0e7ba70e7ef177"} err="failed to get container status \"471a41f1e3af2d2e018072cb11709b8f8fdf6a7983607e69ab0e7ba70e7ef177\": rpc error: code = NotFound desc = could not find container \"471a41f1e3af2d2e018072cb11709b8f8fdf6a7983607e69ab0e7ba70e7ef177\": container with ID starting with 471a41f1e3af2d2e018072cb11709b8f8fdf6a7983607e69ab0e7ba70e7ef177 not found: ID does not exist" Oct 30 02:20:32.120228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:32.120164 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" podStartSLOduration=2.120146431 podStartE2EDuration="2.120146431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:20:32.117963804 +0000 UTC m=+3061.696754886" watchObservedRunningTime="2026-10-30 02:20:32.120146431 +0000 UTC m=+3061.698937503" Oct 30 02:20:32.153323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:32.153286 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g"] Oct 30 02:20:32.157672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:32.157643 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-7f8fsf78g"] Oct 30 02:20:32.542904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:32.542862 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4" path="/var/lib/kubelet/pods/b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4/volumes" Oct 30 02:20:32.979063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:32.979026 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:34.569168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:34.569136 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:36.189929 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:36.189890 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:37.764329 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:37.764295 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:38.218211 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:38.218174 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:39.357169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:39.357131 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:40.916718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:40.916672 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:42.430806 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:42.430725 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:42.563634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:42.563551 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:42.728948 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:42.728845 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:42.994902 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:42.994825 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn"] Oct 30 02:20:42.997720 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:42.997695 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" Oct 30 02:20:44.061545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:44.061490 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:45.778556 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:45.778517 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:45.911964 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:45.911923 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh"] Oct 30 02:20:45.912823 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:45.912802 5998 cpu_manager.go:414] "containerMap: removing container" podUID="b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4" containerName="payload-processing" Oct 30 02:20:45.912894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:45.912827 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4" containerName="payload-processing" Oct 30 02:20:45.913035 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:45.913022 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8174b8d-1a7a-4ca2-800f-0f90f1a2e2a4" containerName="payload-processing" Oct 30 02:20:45.919440 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:45.919413 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:45.950710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:45.950675 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh"] Oct 30 02:20:46.018647 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.018606 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.018855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.018654 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.018855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.018798 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.019000 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.018888 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2mj8\" (UniqueName: \"kubernetes.io/projected/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-kube-api-access-p2mj8\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.019000 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.018956 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.019126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.019010 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.019207 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.019191 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.019278 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.019231 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.019349 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.019333 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.120854 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.120743 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.120854 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.120786 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.120854 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.120837 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.121212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.120879 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2mj8\" (UniqueName: \"kubernetes.io/projected/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-kube-api-access-p2mj8\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.121212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.120910 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.121212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.120956 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.121212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.121066 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.121212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.121112 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.121212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.121177 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.121554 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.121330 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.121554 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.121424 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.122180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.121968 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.122180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.121997 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.122180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.122052 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.124415 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.124383 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.124722 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.124698 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.143437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.143406 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2mj8\" (UniqueName: \"kubernetes.io/projected/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-kube-api-access-p2mj8\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.144517 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.144492 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-ssskh\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.238328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.238291 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:46.576070 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.575956 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh"] Oct 30 02:20:46.578536 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:20:46.578508 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7478a4a2_ef1a_4d42_9c11_4ca61c11ac70.slice/crio-ea8cda631762817050b2f4606a8c28b62cf6f57a885bc6b6ccb044eceed0e636 WatchSource:0}: Error finding container ea8cda631762817050b2f4606a8c28b62cf6f57a885bc6b6ccb044eceed0e636: Status 404 returned error can't find the container with id ea8cda631762817050b2f4606a8c28b62cf6f57a885bc6b6ccb044eceed0e636 Oct 30 02:20:46.581890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.581847 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:20:46.582008 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.581942 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:20:46.582008 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:46.581986 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:20:47.239111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:47.237458 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" event={"ID":"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70","Type":"ContainerStarted","Data":"84c0ff3b7967a3cfda918ac128b2b0c927359ef6f7a01a5a49fc77efdb625b73"} Oct 30 02:20:47.239111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:47.237523 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" event={"ID":"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70","Type":"ContainerStarted","Data":"ea8cda631762817050b2f4606a8c28b62cf6f57a885bc6b6ccb044eceed0e636"} Oct 30 02:20:47.239111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:47.238977 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:47.241526 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:47.241497 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ssskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 02:20:47.241675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:47.241553 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" podUID="7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 02:20:47.297044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:47.296835 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" podStartSLOduration=2.296815903 podStartE2EDuration="2.296815903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:20:47.293403923 +0000 UTC m=+3076.872195003" watchObservedRunningTime="2026-10-30 02:20:47.296815903 +0000 UTC m=+3076.875606986" Oct 30 02:20:47.325201 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:47.323669 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:47.995755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:47.995707 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" podUID="525eb671-a86f-4f82-9bc8-7b53d2801c91" containerName="payload-processing" containerID="cri-o://ac9303f0772b1c8a33217ad7d1c13616205564c9f0997488c6b0b0cc150146f4" gracePeriod=25 Oct 30 02:20:48.250765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:48.249699 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ssskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 02:20:48.250765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:48.249768 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" podUID="7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 02:20:48.264442 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:48.264209 5998 generic.go:356] "Generic (PLEG): container finished" podID="525eb671-a86f-4f82-9bc8-7b53d2801c91" containerID="ac9303f0772b1c8a33217ad7d1c13616205564c9f0997488c6b0b0cc150146f4" exitCode=0 Oct 30 02:20:48.266124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:48.264941 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" event={"ID":"525eb671-a86f-4f82-9bc8-7b53d2801c91","Type":"ContainerDied","Data":"ac9303f0772b1c8a33217ad7d1c13616205564c9f0997488c6b0b0cc150146f4"} Oct 30 02:20:48.348716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:48.348687 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" Oct 30 02:20:48.456943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:48.456903 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/525eb671-a86f-4f82-9bc8-7b53d2801c91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/525eb671-a86f-4f82-9bc8-7b53d2801c91-plugins-config-volume\") pod \"525eb671-a86f-4f82-9bc8-7b53d2801c91\" (UID: \"525eb671-a86f-4f82-9bc8-7b53d2801c91\") " Oct 30 02:20:48.457162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:48.457114 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525eb671-a86f-4f82-9bc8-7b53d2801c91-kube-api-access-xwglz\" (UniqueName: \"kubernetes.io/projected/525eb671-a86f-4f82-9bc8-7b53d2801c91-kube-api-access-xwglz\") pod \"525eb671-a86f-4f82-9bc8-7b53d2801c91\" (UID: \"525eb671-a86f-4f82-9bc8-7b53d2801c91\") " Oct 30 02:20:48.457260 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:48.457243 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525eb671-a86f-4f82-9bc8-7b53d2801c91-provider-credentials\" (UniqueName: \"kubernetes.io/projected/525eb671-a86f-4f82-9bc8-7b53d2801c91-provider-credentials\") pod \"525eb671-a86f-4f82-9bc8-7b53d2801c91\" (UID: \"525eb671-a86f-4f82-9bc8-7b53d2801c91\") " Oct 30 02:20:48.457336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:48.457288 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/525eb671-a86f-4f82-9bc8-7b53d2801c91-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/525eb671-a86f-4f82-9bc8-7b53d2801c91-routing-overlay-volume\") pod \"525eb671-a86f-4f82-9bc8-7b53d2801c91\" (UID: \"525eb671-a86f-4f82-9bc8-7b53d2801c91\") " Oct 30 02:20:48.457441 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:48.457415 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525eb671-a86f-4f82-9bc8-7b53d2801c91-plugins-config-volume" pod "525eb671-a86f-4f82-9bc8-7b53d2801c91" (UID: "525eb671-a86f-4f82-9bc8-7b53d2801c91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:48.457709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:48.457682 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525eb671-a86f-4f82-9bc8-7b53d2801c91-routing-overlay-volume" pod "525eb671-a86f-4f82-9bc8-7b53d2801c91" (UID: "525eb671-a86f-4f82-9bc8-7b53d2801c91"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:48.457936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:48.457918 5998 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/525eb671-a86f-4f82-9bc8-7b53d2801c91-routing-overlay-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:20:48.458036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:48.457944 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/525eb671-a86f-4f82-9bc8-7b53d2801c91-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:20:48.459917 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:48.459893 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525eb671-a86f-4f82-9bc8-7b53d2801c91-provider-credentials" pod "525eb671-a86f-4f82-9bc8-7b53d2801c91" (UID: "525eb671-a86f-4f82-9bc8-7b53d2801c91"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:48.460031 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:48.459924 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525eb671-a86f-4f82-9bc8-7b53d2801c91-kube-api-access-xwglz" pod "525eb671-a86f-4f82-9bc8-7b53d2801c91" (UID: "525eb671-a86f-4f82-9bc8-7b53d2801c91"). InnerVolumeSpecName "kube-api-access-xwglz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:48.560164 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:48.560134 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwglz\" (UniqueName: \"kubernetes.io/projected/525eb671-a86f-4f82-9bc8-7b53d2801c91-kube-api-access-xwglz\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:20:48.560369 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:48.560352 5998 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/525eb671-a86f-4f82-9bc8-7b53d2801c91-provider-credentials\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:20:48.837195 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:48.836671 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:49.240111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:49.238838 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ssskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 02:20:49.240111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:49.238906 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" podUID="7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 02:20:49.273562 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:49.273528 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" Oct 30 02:20:49.273562 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:49.273539 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn" event={"ID":"525eb671-a86f-4f82-9bc8-7b53d2801c91","Type":"ContainerDied","Data":"039a05d610daddc33e65bdbcd740a645ab0ab262252e77919cea83d7d59af687"} Oct 30 02:20:49.274062 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:49.273641 5998 scope.go:122] "RemoveContainer" containerID="ac9303f0772b1c8a33217ad7d1c13616205564c9f0997488c6b0b0cc150146f4" Oct 30 02:20:49.334291 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:49.334254 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn"] Oct 30 02:20:49.340497 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:49.340470 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a0fc73/payload-processing-external-model-e2e-extmod-a-a0fc73-8477hdmfn"] Oct 30 02:20:49.719114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:49.718598 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:50.244754 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:50.244709 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ssskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 02:20:50.244978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:50.244781 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" podUID="7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 02:20:50.417186 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:50.417140 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:50.544004 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:50.543917 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="525eb671-a86f-4f82-9bc8-7b53d2801c91" path="/var/lib/kubelet/pods/525eb671-a86f-4f82-9bc8-7b53d2801c91/volumes" Oct 30 02:20:51.239394 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:51.238846 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ssskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 02:20:51.239394 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:51.238921 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" podUID="7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 02:20:51.972144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:51.972103 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:52.239012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:52.238910 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ssskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 02:20:52.239012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:52.238968 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" podUID="7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 02:20:53.238848 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:53.238807 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ssskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 02:20:53.239444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:53.238871 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" podUID="7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 02:20:53.483066 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:53.483024 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:54.239376 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:54.238915 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ssskh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body= Oct 30 02:20:54.239376 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:54.238987 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" podUID="7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" Oct 30 02:20:54.994974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:54.994917 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:55.225157 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:20:55.225111 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:20:55.246065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:55.245985 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:55.246573 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:55.246519 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:55.247352 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:55.247326 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:20:56.660587 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:56.660495 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:58.159903 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:58.159865 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:20:59.705007 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:20:59.704970 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:00.975403 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:00.975366 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft"] Oct 30 02:21:00.976251 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:00.976222 5998 cpu_manager.go:414] "containerMap: removing container" podUID="525eb671-a86f-4f82-9bc8-7b53d2801c91" containerName="payload-processing" Oct 30 02:21:00.976445 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:00.976423 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525eb671-a86f-4f82-9bc8-7b53d2801c91" containerName="payload-processing" Oct 30 02:21:00.976765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:00.976746 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="525eb671-a86f-4f82-9bc8-7b53d2801c91" containerName="payload-processing" Oct 30 02:21:00.986057 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:00.986031 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.204154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.199596 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft"] Oct 30 02:21:01.228417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.219997 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.228417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.220164 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.228417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.220251 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.228417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.220292 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.228417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.220314 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.228417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.220349 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.228417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.220381 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mx55\" (UniqueName: \"kubernetes.io/projected/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-kube-api-access-8mx55\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.228417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.220459 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.228417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.220496 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.436166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.321579 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.436166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.362203 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.436166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.362254 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.436166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.362301 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.436166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.362342 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mx55\" (UniqueName: \"kubernetes.io/projected/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-kube-api-access-8mx55\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.436166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.362461 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.436166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.362503 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.436166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.362606 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.436166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.362754 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.436166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.327822 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.436166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.363694 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.518248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.491191 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.518248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.500910 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.518248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.501613 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.639125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.638928 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.641302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.641240 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.643991 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.643965 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mx55\" (UniqueName: \"kubernetes.io/projected/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-kube-api-access-8mx55\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.687120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.685998 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-jhqft\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.748145 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.748041 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:01.852774 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:01.845698 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:02.193129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:02.192484 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft"] Oct 30 02:21:02.196302 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:21:02.196276 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf186c2e9_3b5d_48e7_b649_5bcd5cd38ce6.slice/crio-2d45619b585edc7253fdefd14779aadb9afbc887f093a000316ffef165c0e055 WatchSource:0}: Error finding container 2d45619b585edc7253fdefd14779aadb9afbc887f093a000316ffef165c0e055: Status 404 returned error can't find the container with id 2d45619b585edc7253fdefd14779aadb9afbc887f093a000316ffef165c0e055 Oct 30 02:21:02.200376 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:02.200338 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:21:02.200479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:02.200433 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:21:02.200525 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:02.200496 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:21:02.531170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:02.529658 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" event={"ID":"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6","Type":"ContainerStarted","Data":"d6b4b15fa2b40704aff9c0cf6366a1d36fa746b3e2b401f9bac47d41b9e0ff5e"} Oct 30 02:21:02.531170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:02.529734 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" event={"ID":"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6","Type":"ContainerStarted","Data":"2d45619b585edc7253fdefd14779aadb9afbc887f093a000316ffef165c0e055"} Oct 30 02:21:02.578410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:02.577266 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" podStartSLOduration=2.577247293 podStartE2EDuration="2.577247293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:21:02.57523495 +0000 UTC m=+3092.154026032" watchObservedRunningTime="2026-10-30 02:21:02.577247293 +0000 UTC m=+3092.156038371" Oct 30 02:21:02.759115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:02.755293 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:03.759123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:03.755967 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jhqft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 02:21:03.759123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:03.756041 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" podUID="f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 02:21:03.786112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:03.785767 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:04.764115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:04.762475 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jhqft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 02:21:04.764115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:04.762538 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" podUID="f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 02:21:05.059880 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.058264 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2"] Oct 30 02:21:05.135761 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.135711 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2"] Oct 30 02:21:05.137719 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.137607 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2" Oct 30 02:21:05.232120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.216344 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:21:05.232120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.216460 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:21:05.232120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.216522 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:21:05.263174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.247392 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjcp8\" (UniqueName: \"kubernetes.io/projected/d703a3a6-1722-48d3-bbee-5913ccd2f487-kube-api-access-vjcp8\") pod \"maas-api-revoke-sub-57e5090b-fwfs2\" (UID: \"d703a3a6-1722-48d3-bbee-5913ccd2f487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2" Oct 30 02:21:05.263174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.247511 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d703a3a6-1722-48d3-bbee-5913ccd2f487-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57e5090b-fwfs2\" (UID: \"d703a3a6-1722-48d3-bbee-5913ccd2f487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2" Oct 30 02:21:05.263174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.247675 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d703a3a6-1722-48d3-bbee-5913ccd2f487-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57e5090b-fwfs2\" (UID: \"d703a3a6-1722-48d3-bbee-5913ccd2f487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2" Oct 30 02:21:05.311561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.311441 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 02:21:05.334819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.334778 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:21:05.380212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.380177 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2"] Oct 30 02:21:05.537300 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.537266 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d703a3a6-1722-48d3-bbee-5913ccd2f487-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57e5090b-fwfs2\" (UID: \"d703a3a6-1722-48d3-bbee-5913ccd2f487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2" Oct 30 02:21:05.548035 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.548005 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjcp8\" (UniqueName: \"kubernetes.io/projected/d703a3a6-1722-48d3-bbee-5913ccd2f487-kube-api-access-vjcp8\") pod \"maas-api-revoke-sub-57e5090b-fwfs2\" (UID: \"d703a3a6-1722-48d3-bbee-5913ccd2f487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2" Oct 30 02:21:05.563019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.562934 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d703a3a6-1722-48d3-bbee-5913ccd2f487-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57e5090b-fwfs2\" (UID: \"d703a3a6-1722-48d3-bbee-5913ccd2f487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2" Oct 30 02:21:05.563401 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.563285 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d703a3a6-1722-48d3-bbee-5913ccd2f487-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57e5090b-fwfs2\" (UID: \"d703a3a6-1722-48d3-bbee-5913ccd2f487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2" Oct 30 02:21:05.631274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.606399 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d703a3a6-1722-48d3-bbee-5913ccd2f487-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57e5090b-fwfs2\" (UID: \"d703a3a6-1722-48d3-bbee-5913ccd2f487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2" Oct 30 02:21:05.631274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.612340 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2"] Oct 30 02:21:05.631274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.613462 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2" Oct 30 02:21:05.665113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.663825 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjcp8\" (UniqueName: \"kubernetes.io/projected/d703a3a6-1722-48d3-bbee-5913ccd2f487-kube-api-access-vjcp8\") pod \"maas-api-revoke-sub-57e5090b-fwfs2\" (UID: \"d703a3a6-1722-48d3-bbee-5913ccd2f487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2" Oct 30 02:21:05.788120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.785320 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebf1325-84b3-47d8-a096-af69c5b55aec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-374ef512-7rwv2\" (UID: \"4ebf1325-84b3-47d8-a096-af69c5b55aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2" Oct 30 02:21:05.788120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.785486 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebf1325-84b3-47d8-a096-af69c5b55aec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-374ef512-7rwv2\" (UID: \"4ebf1325-84b3-47d8-a096-af69c5b55aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2" Oct 30 02:21:05.788120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.785772 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvcsf\" (UniqueName: \"kubernetes.io/projected/4ebf1325-84b3-47d8-a096-af69c5b55aec-kube-api-access-xvcsf\") pod \"maas-api-revoke-sub-374ef512-7rwv2\" (UID: \"4ebf1325-84b3-47d8-a096-af69c5b55aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2" Oct 30 02:21:05.808285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.808242 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jhqft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 02:21:05.808601 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.808567 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" podUID="f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 02:21:05.838210 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.837798 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:05.898152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.889377 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebf1325-84b3-47d8-a096-af69c5b55aec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-374ef512-7rwv2\" (UID: \"4ebf1325-84b3-47d8-a096-af69c5b55aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2" Oct 30 02:21:05.898152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.890423 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebf1325-84b3-47d8-a096-af69c5b55aec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-374ef512-7rwv2\" (UID: \"4ebf1325-84b3-47d8-a096-af69c5b55aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2" Oct 30 02:21:05.898152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.890648 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebf1325-84b3-47d8-a096-af69c5b55aec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-374ef512-7rwv2\" (UID: \"4ebf1325-84b3-47d8-a096-af69c5b55aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2" Oct 30 02:21:05.898152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.891425 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvcsf\" (UniqueName: \"kubernetes.io/projected/4ebf1325-84b3-47d8-a096-af69c5b55aec-kube-api-access-xvcsf\") pod \"maas-api-revoke-sub-374ef512-7rwv2\" (UID: \"4ebf1325-84b3-47d8-a096-af69c5b55aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2" Oct 30 02:21:05.915126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.913959 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2" Oct 30 02:21:05.945197 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.917488 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvcsf\" (UniqueName: \"kubernetes.io/projected/4ebf1325-84b3-47d8-a096-af69c5b55aec-kube-api-access-xvcsf\") pod \"maas-api-revoke-sub-374ef512-7rwv2\" (UID: \"4ebf1325-84b3-47d8-a096-af69c5b55aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2" Oct 30 02:21:05.945197 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.933643 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebf1325-84b3-47d8-a096-af69c5b55aec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-374ef512-7rwv2\" (UID: \"4ebf1325-84b3-47d8-a096-af69c5b55aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2" Oct 30 02:21:05.994368 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:05.994313 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:21:06.123311 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:06.120988 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:21:06.123311 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:06.121424 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 02:21:06.184596 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:06.184304 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2" Oct 30 02:21:06.692226 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:06.692198 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2"] Oct 30 02:21:06.770547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:06.770298 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jhqft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 02:21:06.770547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:06.770352 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" podUID="f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 02:21:06.772116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:06.772069 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2"] Oct 30 02:21:06.875844 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:06.875801 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2" event={"ID":"d703a3a6-1722-48d3-bbee-5913ccd2f487","Type":"ContainerStarted","Data":"01bb0a906a9c894fbbf41aef648f2d8b046ec33256062a3ca808247a587ab0b4"} Oct 30 02:21:06.878721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:06.878691 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2" event={"ID":"4ebf1325-84b3-47d8-a096-af69c5b55aec","Type":"ContainerStarted","Data":"c79ca5abf19ed94971897b4476d198f31ab82bc37b42d12f064dde3ae0c6cd2f"} Oct 30 02:21:07.528770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:07.528674 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:07.760116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:07.751510 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jhqft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 02:21:07.760116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:07.751595 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" podUID="f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 02:21:07.891251 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:07.891157 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2" event={"ID":"d703a3a6-1722-48d3-bbee-5913ccd2f487","Type":"ContainerStarted","Data":"c0e3ab1ae466790562d93f3af408abc9f39f3f38de0e1956c81ef9c635d6f441"} Oct 30 02:21:07.893305 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:07.893274 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2" event={"ID":"4ebf1325-84b3-47d8-a096-af69c5b55aec","Type":"ContainerStarted","Data":"d6e94e1a5b2feb5347978c60628d517d083cdff6326b8cef9195a2740af0a2e6"} Oct 30 02:21:07.970115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:07.967012 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2" podStartSLOduration=3.966991034 podStartE2EDuration="3.966991034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:21:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:21:07.936763929 +0000 UTC m=+3097.515555014" watchObservedRunningTime="2026-10-30 02:21:07.966991034 +0000 UTC m=+3097.545782116" Oct 30 02:21:08.749724 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:08.749680 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jhqft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 02:21:08.749927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:08.749752 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" podUID="f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 02:21:08.928700 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:08.927791 5998 generic.go:356] "Generic (PLEG): container finished" podID="4ebf1325-84b3-47d8-a096-af69c5b55aec" containerID="d6e94e1a5b2feb5347978c60628d517d083cdff6326b8cef9195a2740af0a2e6" exitCode=0 Oct 30 02:21:08.928700 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:08.928149 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2" event={"ID":"4ebf1325-84b3-47d8-a096-af69c5b55aec","Type":"ContainerDied","Data":"d6e94e1a5b2feb5347978c60628d517d083cdff6326b8cef9195a2740af0a2e6"} Oct 30 02:21:08.931502 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:08.931472 5998 generic.go:356] "Generic (PLEG): container finished" podID="d703a3a6-1722-48d3-bbee-5913ccd2f487" containerID="c0e3ab1ae466790562d93f3af408abc9f39f3f38de0e1956c81ef9c635d6f441" exitCode=0 Oct 30 02:21:08.931766 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:08.931744 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2" event={"ID":"d703a3a6-1722-48d3-bbee-5913ccd2f487","Type":"ContainerDied","Data":"c0e3ab1ae466790562d93f3af408abc9f39f3f38de0e1956c81ef9c635d6f441"} Oct 30 02:21:09.261892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:09.261801 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:09.749638 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:09.749597 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jhqft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 02:21:09.749842 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:09.749684 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" podUID="f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 02:21:10.192141 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.192075 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2" Oct 30 02:21:10.196265 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.196237 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2" Oct 30 02:21:10.218136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.214603 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d703a3a6-1722-48d3-bbee-5913ccd2f487-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d703a3a6-1722-48d3-bbee-5913ccd2f487-maas-api-openshift-ca\") pod \"d703a3a6-1722-48d3-bbee-5913ccd2f487\" (UID: \"d703a3a6-1722-48d3-bbee-5913ccd2f487\") " Oct 30 02:21:10.218136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.214810 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d703a3a6-1722-48d3-bbee-5913ccd2f487-kube-api-access-vjcp8\" (UniqueName: \"kubernetes.io/projected/d703a3a6-1722-48d3-bbee-5913ccd2f487-kube-api-access-vjcp8\") pod \"d703a3a6-1722-48d3-bbee-5913ccd2f487\" (UID: \"d703a3a6-1722-48d3-bbee-5913ccd2f487\") " Oct 30 02:21:10.218136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.214990 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d703a3a6-1722-48d3-bbee-5913ccd2f487-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d703a3a6-1722-48d3-bbee-5913ccd2f487-maas-api-xks-ca\") pod \"d703a3a6-1722-48d3-bbee-5913ccd2f487\" (UID: \"d703a3a6-1722-48d3-bbee-5913ccd2f487\") " Oct 30 02:21:10.218136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.217930 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d703a3a6-1722-48d3-bbee-5913ccd2f487-maas-api-openshift-ca" pod "d703a3a6-1722-48d3-bbee-5913ccd2f487" (UID: "d703a3a6-1722-48d3-bbee-5913ccd2f487"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:10.230382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.227068 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d703a3a6-1722-48d3-bbee-5913ccd2f487-kube-api-access-vjcp8" pod "d703a3a6-1722-48d3-bbee-5913ccd2f487" (UID: "d703a3a6-1722-48d3-bbee-5913ccd2f487"). InnerVolumeSpecName "kube-api-access-vjcp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:10.266706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.266667 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d703a3a6-1722-48d3-bbee-5913ccd2f487-maas-api-xks-ca" pod "d703a3a6-1722-48d3-bbee-5913ccd2f487" (UID: "d703a3a6-1722-48d3-bbee-5913ccd2f487"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:21:10.325113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.316935 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ebf1325-84b3-47d8-a096-af69c5b55aec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebf1325-84b3-47d8-a096-af69c5b55aec-maas-api-openshift-ca\") pod \"4ebf1325-84b3-47d8-a096-af69c5b55aec\" (UID: \"4ebf1325-84b3-47d8-a096-af69c5b55aec\") " Oct 30 02:21:10.325113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.317015 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ebf1325-84b3-47d8-a096-af69c5b55aec-kube-api-access-xvcsf\" (UniqueName: \"kubernetes.io/projected/4ebf1325-84b3-47d8-a096-af69c5b55aec-kube-api-access-xvcsf\") pod \"4ebf1325-84b3-47d8-a096-af69c5b55aec\" (UID: \"4ebf1325-84b3-47d8-a096-af69c5b55aec\") " Oct 30 02:21:10.325113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.317282 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ebf1325-84b3-47d8-a096-af69c5b55aec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebf1325-84b3-47d8-a096-af69c5b55aec-maas-api-xks-ca\") pod \"4ebf1325-84b3-47d8-a096-af69c5b55aec\" (UID: \"4ebf1325-84b3-47d8-a096-af69c5b55aec\") " Oct 30 02:21:10.325113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.317691 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d703a3a6-1722-48d3-bbee-5913ccd2f487-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:21:10.325113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.317705 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d703a3a6-1722-48d3-bbee-5913ccd2f487-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:21:10.325113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.317714 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjcp8\" (UniqueName: \"kubernetes.io/projected/d703a3a6-1722-48d3-bbee-5913ccd2f487-kube-api-access-vjcp8\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:21:10.325113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.318519 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebf1325-84b3-47d8-a096-af69c5b55aec-maas-api-openshift-ca" pod "4ebf1325-84b3-47d8-a096-af69c5b55aec" (UID: "4ebf1325-84b3-47d8-a096-af69c5b55aec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:10.325113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.324483 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ebf1325-84b3-47d8-a096-af69c5b55aec-maas-api-xks-ca" pod "4ebf1325-84b3-47d8-a096-af69c5b55aec" (UID: "4ebf1325-84b3-47d8-a096-af69c5b55aec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:21:10.349615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.326724 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebf1325-84b3-47d8-a096-af69c5b55aec-kube-api-access-xvcsf" pod "4ebf1325-84b3-47d8-a096-af69c5b55aec" (UID: "4ebf1325-84b3-47d8-a096-af69c5b55aec"). InnerVolumeSpecName "kube-api-access-xvcsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:10.419608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.419560 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ebf1325-84b3-47d8-a096-af69c5b55aec-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:21:10.419608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.419609 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ebf1325-84b3-47d8-a096-af69c5b55aec-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:21:10.419908 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.419628 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvcsf\" (UniqueName: \"kubernetes.io/projected/4ebf1325-84b3-47d8-a096-af69c5b55aec-kube-api-access-xvcsf\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:21:10.749333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.749293 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jhqft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Oct 30 02:21:10.749487 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.749362 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" podUID="f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Oct 30 02:21:10.951862 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.951826 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2" Oct 30 02:21:10.952139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.951857 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2" event={"ID":"4ebf1325-84b3-47d8-a096-af69c5b55aec","Type":"ContainerDied","Data":"c79ca5abf19ed94971897b4476d198f31ab82bc37b42d12f064dde3ae0c6cd2f"} Oct 30 02:21:10.952139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.951918 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c79ca5abf19ed94971897b4476d198f31ab82bc37b42d12f064dde3ae0c6cd2f" Oct 30 02:21:10.953674 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.953646 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2" event={"ID":"d703a3a6-1722-48d3-bbee-5913ccd2f487","Type":"ContainerDied","Data":"01bb0a906a9c894fbbf41aef648f2d8b046ec33256062a3ca808247a587ab0b4"} Oct 30 02:21:10.953791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.953680 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2" Oct 30 02:21:10.953791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:10.953696 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01bb0a906a9c894fbbf41aef648f2d8b046ec33256062a3ca808247a587ab0b4" Oct 30 02:21:11.238154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:11.236778 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:12.174844 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:12.174815 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:12.175174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:12.175147 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:12.176454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:12.176429 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:21:12.922114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:12.920686 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:13.984217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:13.984182 5998 generic.go:356] "Generic (PLEG): container finished" podID="b13ac864-0406-4db5-b911-2851f86df2b6" containerID="81cb1590c843dc22421c710b134b517b7ca49efb367815ff446e77a825311f55" exitCode=0 Oct 30 02:21:13.984717 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:13.984255 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" event={"ID":"b13ac864-0406-4db5-b911-2851f86df2b6","Type":"ContainerDied","Data":"81cb1590c843dc22421c710b134b517b7ca49efb367815ff446e77a825311f55"} Oct 30 02:21:13.984717 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:13.984688 5998 scope.go:122] "RemoveContainer" containerID="81cb1590c843dc22421c710b134b517b7ca49efb367815ff446e77a825311f55" Oct 30 02:21:13.986383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:13.986357 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:21:13.986473 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:13.986422 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:21:13.986473 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:13.986464 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:21:14.456064 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:14.456028 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:14.585060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:14.585033 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:21:14.993792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:14.993749 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" event={"ID":"b13ac864-0406-4db5-b911-2851f86df2b6","Type":"ContainerStarted","Data":"d31176aa8e60ad566c31291b46dd242581dbf1080e275625f91f4501017a382c"} Oct 30 02:21:15.714116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:15.712806 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:21:15.715120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:15.714560 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 02:21:15.715120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:15.714620 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" podUID="b13ac864-0406-4db5-b911-2851f86df2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 02:21:16.069172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.066422 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7"] Oct 30 02:21:16.069172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.067402 5998 cpu_manager.go:414] "containerMap: removing container" podUID="d703a3a6-1722-48d3-bbee-5913ccd2f487" containerName="revoke-keys" Oct 30 02:21:16.069172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.067421 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d703a3a6-1722-48d3-bbee-5913ccd2f487" containerName="revoke-keys" Oct 30 02:21:16.069172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.067438 5998 cpu_manager.go:414] "containerMap: removing container" podUID="4ebf1325-84b3-47d8-a096-af69c5b55aec" containerName="revoke-keys" Oct 30 02:21:16.069172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.067443 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebf1325-84b3-47d8-a096-af69c5b55aec" containerName="revoke-keys" Oct 30 02:21:16.069172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.067659 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="d703a3a6-1722-48d3-bbee-5913ccd2f487" containerName="revoke-keys" Oct 30 02:21:16.069172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.067673 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ebf1325-84b3-47d8-a096-af69c5b55aec" containerName="revoke-keys" Oct 30 02:21:16.072029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.071699 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.079769 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.079718 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7"] Oct 30 02:21:16.143527 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.143472 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qb9v\" (UniqueName: \"kubernetes.io/projected/9782ca9d-bd87-415c-bad8-e080814048e7-kube-api-access-4qb9v\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.143977 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.143949 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.144231 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.144211 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.144475 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.144455 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.144633 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.144614 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9782ca9d-bd87-415c-bad8-e080814048e7-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.144859 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.144842 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.145033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.145010 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.145170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.145152 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9782ca9d-bd87-415c-bad8-e080814048e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.145371 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.145354 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9782ca9d-bd87-415c-bad8-e080814048e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.252116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.247059 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qb9v\" (UniqueName: \"kubernetes.io/projected/9782ca9d-bd87-415c-bad8-e080814048e7-kube-api-access-4qb9v\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.252116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.247182 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.252116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.247241 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.252116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.247303 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.252116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.247337 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9782ca9d-bd87-415c-bad8-e080814048e7-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.252116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.247414 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.252116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.247478 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.252116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.247502 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9782ca9d-bd87-415c-bad8-e080814048e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.252116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.247568 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9782ca9d-bd87-415c-bad8-e080814048e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.252116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.248401 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9782ca9d-bd87-415c-bad8-e080814048e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.252116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.249558 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.252116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.249795 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.252116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.250028 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.256210 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.253473 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:16.259252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.259192 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9782ca9d-bd87-415c-bad8-e080814048e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.261442 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.261136 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.305827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.305790 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9782ca9d-bd87-415c-bad8-e080814048e7-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.490153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.433011 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qb9v\" (UniqueName: \"kubernetes.io/projected/9782ca9d-bd87-415c-bad8-e080814048e7-kube-api-access-4qb9v\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.490153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.433266 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-fpsp7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.490153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.447063 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:16.490153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.447862 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:16.577203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.577121 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:16.709669 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.709203 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 02:21:16.709669 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:16.709265 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" podUID="b13ac864-0406-4db5-b911-2851f86df2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 02:21:17.067430 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:17.063987 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7"] Oct 30 02:21:17.080399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:17.077828 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:21:17.080399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:17.077928 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:21:17.080399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:17.077970 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:21:17.717416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:17.717378 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 02:21:17.717713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:17.717683 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" podUID="b13ac864-0406-4db5-b911-2851f86df2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 02:21:18.083572 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:18.082295 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" event={"ID":"9782ca9d-bd87-415c-bad8-e080814048e7","Type":"ContainerStarted","Data":"d0d222cc228a1afd865c05be868c3014eb31cfafa83a82c0e44d46117f3546a4"} Oct 30 02:21:18.083572 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:18.082367 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" event={"ID":"9782ca9d-bd87-415c-bad8-e080814048e7","Type":"ContainerStarted","Data":"cc03c13477795081edcde2c16921ff762181e0fbf6dd7c11ee510eff2f9f8c56"} Oct 30 02:21:18.174359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:18.174267 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" podStartSLOduration=2.174245346 podStartE2EDuration="2.174245346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:21:18.159554402 +0000 UTC m=+3107.738345496" watchObservedRunningTime="2026-10-30 02:21:18.174245346 +0000 UTC m=+3107.753036428" Oct 30 02:21:18.261799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:18.261763 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:18.298337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:18.290879 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s"] Oct 30 02:21:18.457323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:18.457293 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:18.586947 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:18.586912 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fpsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 02:21:18.587229 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:18.587194 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" podUID="9782ca9d-bd87-415c-bad8-e080814048e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 02:21:18.721763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:18.721272 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 02:21:18.721763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:18.721328 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" podUID="b13ac864-0406-4db5-b911-2851f86df2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 02:21:18.781118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:18.777603 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:19.027172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:19.020379 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:19.455124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:19.454624 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fpsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 02:21:19.455124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:19.454720 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" podUID="9782ca9d-bd87-415c-bad8-e080814048e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 02:21:19.714209 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:19.711621 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 02:21:19.714209 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:19.711702 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" podUID="b13ac864-0406-4db5-b911-2851f86df2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 02:21:20.100174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:20.099955 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:20.458126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:20.451171 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fpsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 02:21:20.458126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:20.451234 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" podUID="9782ca9d-bd87-415c-bad8-e080814048e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 02:21:20.725175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:20.712675 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 02:21:20.725175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:20.714056 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" podUID="b13ac864-0406-4db5-b911-2851f86df2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 02:21:20.988191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:20.988084 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:21.450920 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:21.450615 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fpsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 02:21:21.450920 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:21.450711 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" podUID="9782ca9d-bd87-415c-bad8-e080814048e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 02:21:21.718624 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:21.718523 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 02:21:21.719249 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:21.719199 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" podUID="b13ac864-0406-4db5-b911-2851f86df2b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 02:21:22.449593 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:22.449557 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fpsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 02:21:22.449784 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:22.449644 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" podUID="9782ca9d-bd87-415c-bad8-e080814048e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 02:21:22.732252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:22.732161 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:21:22.732680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:22.732480 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:21:22.733318 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:22.733294 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:21:22.840203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:22.840164 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:23.291485 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:23.291446 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" podUID="428ef227-997d-4210-a1ef-de5cf92fc1a5" containerName="payload-processing" containerID="cri-o://bd0d0865e11411c8f0691e152e157874560fc23aeb0da57b670e8dc01603bf6c" gracePeriod=25 Oct 30 02:21:23.449423 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:23.449389 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fpsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 02:21:23.449616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:23.449450 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" podUID="9782ca9d-bd87-415c-bad8-e080814048e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 02:21:23.654937 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:23.654912 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" Oct 30 02:21:23.850368 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:23.850331 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428ef227-997d-4210-a1ef-de5cf92fc1a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428ef227-997d-4210-a1ef-de5cf92fc1a5-plugins-config-volume\") pod \"428ef227-997d-4210-a1ef-de5cf92fc1a5\" (UID: \"428ef227-997d-4210-a1ef-de5cf92fc1a5\") " Oct 30 02:21:23.850884 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:23.850454 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428ef227-997d-4210-a1ef-de5cf92fc1a5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428ef227-997d-4210-a1ef-de5cf92fc1a5-routing-overlay-volume\") pod \"428ef227-997d-4210-a1ef-de5cf92fc1a5\" (UID: \"428ef227-997d-4210-a1ef-de5cf92fc1a5\") " Oct 30 02:21:23.850884 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:23.850495 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428ef227-997d-4210-a1ef-de5cf92fc1a5-kube-api-access-7kfjp\" (UniqueName: \"kubernetes.io/projected/428ef227-997d-4210-a1ef-de5cf92fc1a5-kube-api-access-7kfjp\") pod \"428ef227-997d-4210-a1ef-de5cf92fc1a5\" (UID: \"428ef227-997d-4210-a1ef-de5cf92fc1a5\") " Oct 30 02:21:23.850884 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:23.850556 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428ef227-997d-4210-a1ef-de5cf92fc1a5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/428ef227-997d-4210-a1ef-de5cf92fc1a5-provider-credentials\") pod \"428ef227-997d-4210-a1ef-de5cf92fc1a5\" (UID: \"428ef227-997d-4210-a1ef-de5cf92fc1a5\") " Oct 30 02:21:23.851055 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:23.850877 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428ef227-997d-4210-a1ef-de5cf92fc1a5-plugins-config-volume" pod "428ef227-997d-4210-a1ef-de5cf92fc1a5" (UID: "428ef227-997d-4210-a1ef-de5cf92fc1a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:23.851055 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:23.850909 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428ef227-997d-4210-a1ef-de5cf92fc1a5-routing-overlay-volume" pod "428ef227-997d-4210-a1ef-de5cf92fc1a5" (UID: "428ef227-997d-4210-a1ef-de5cf92fc1a5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:23.851338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:23.851319 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428ef227-997d-4210-a1ef-de5cf92fc1a5-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:21:23.851402 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:23.851343 5998 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/428ef227-997d-4210-a1ef-de5cf92fc1a5-routing-overlay-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:21:23.853539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:23.853451 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428ef227-997d-4210-a1ef-de5cf92fc1a5-provider-credentials" pod "428ef227-997d-4210-a1ef-de5cf92fc1a5" (UID: "428ef227-997d-4210-a1ef-de5cf92fc1a5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:23.853539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:23.853479 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428ef227-997d-4210-a1ef-de5cf92fc1a5-kube-api-access-7kfjp" pod "428ef227-997d-4210-a1ef-de5cf92fc1a5" (UID: "428ef227-997d-4210-a1ef-de5cf92fc1a5"). InnerVolumeSpecName "kube-api-access-7kfjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:23.953055 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:23.953018 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kfjp\" (UniqueName: \"kubernetes.io/projected/428ef227-997d-4210-a1ef-de5cf92fc1a5-kube-api-access-7kfjp\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:21:23.953055 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:23.953051 5998 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/428ef227-997d-4210-a1ef-de5cf92fc1a5-provider-credentials\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:21:24.364879 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:24.364796 5998 generic.go:356] "Generic (PLEG): container finished" podID="428ef227-997d-4210-a1ef-de5cf92fc1a5" containerID="bd0d0865e11411c8f0691e152e157874560fc23aeb0da57b670e8dc01603bf6c" exitCode=0 Oct 30 02:21:24.364879 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:24.364845 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" Oct 30 02:21:24.365112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:24.364882 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" event={"ID":"428ef227-997d-4210-a1ef-de5cf92fc1a5","Type":"ContainerDied","Data":"bd0d0865e11411c8f0691e152e157874560fc23aeb0da57b670e8dc01603bf6c"} Oct 30 02:21:24.365112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:24.364939 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s" event={"ID":"428ef227-997d-4210-a1ef-de5cf92fc1a5","Type":"ContainerDied","Data":"ec5bc21879aea8a5dda2d8c6da024ae6195a16bfec57b9c581206bf47ed9399f"} Oct 30 02:21:24.365112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:24.364968 5998 scope.go:122] "RemoveContainer" containerID="bd0d0865e11411c8f0691e152e157874560fc23aeb0da57b670e8dc01603bf6c" Oct 30 02:21:24.375583 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:24.375564 5998 scope.go:122] "RemoveContainer" containerID="bd0d0865e11411c8f0691e152e157874560fc23aeb0da57b670e8dc01603bf6c" Oct 30 02:21:24.375869 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:21:24.375851 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd0d0865e11411c8f0691e152e157874560fc23aeb0da57b670e8dc01603bf6c\": container with ID starting with bd0d0865e11411c8f0691e152e157874560fc23aeb0da57b670e8dc01603bf6c not found: ID does not exist" containerID="bd0d0865e11411c8f0691e152e157874560fc23aeb0da57b670e8dc01603bf6c" Oct 30 02:21:24.375910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:24.375880 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0d0865e11411c8f0691e152e157874560fc23aeb0da57b670e8dc01603bf6c"} err="failed to get container status \"bd0d0865e11411c8f0691e152e157874560fc23aeb0da57b670e8dc01603bf6c\": rpc error: code = NotFound desc = could not find container \"bd0d0865e11411c8f0691e152e157874560fc23aeb0da57b670e8dc01603bf6c\": container with ID starting with bd0d0865e11411c8f0691e152e157874560fc23aeb0da57b670e8dc01603bf6c not found: ID does not exist" Oct 30 02:21:24.432240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:24.432156 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s"] Oct 30 02:21:24.437045 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:24.437019 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-58fad4/payload-processing-external-model-e2e-extmod-a-58fad4-57c7zn62s"] Oct 30 02:21:24.450168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:24.450132 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fpsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 02:21:24.450331 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:24.450195 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" podUID="9782ca9d-bd87-415c-bad8-e080814048e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 02:21:24.541311 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:24.541279 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428ef227-997d-4210-a1ef-de5cf92fc1a5" path="/var/lib/kubelet/pods/428ef227-997d-4210-a1ef-de5cf92fc1a5/volumes" Oct 30 02:21:24.627119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:24.626995 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:24.992561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:24.992435 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:25.141495 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:25.141448 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:25.333181 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:21:25.330177 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:21:25.454500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:25.454462 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fpsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 02:21:25.454806 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:25.454775 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" podUID="9782ca9d-bd87-415c-bad8-e080814048e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 02:21:26.450017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:26.449979 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fpsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 02:21:26.450581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:26.450045 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" podUID="9782ca9d-bd87-415c-bad8-e080814048e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 02:21:26.618202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:26.618166 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:27.226396 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:27.226358 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:27.391539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:27.391493 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:27.517955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:27.517862 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:28.267755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:28.267707 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:28.268419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:28.268384 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:28.269408 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:28.269384 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:21:28.417977 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:28.417939 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:30.039405 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:30.039367 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:31.139532 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:31.139490 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:31.255717 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:31.255680 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:31.906126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:31.906054 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:32.043111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:32.042996 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:33.240030 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:33.239992 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:33.331324 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:33.331287 5998 scope.go:122] "RemoveContainer" containerID="e72fac36fe6f8e02dec78f560cc2e0227b538576bd0060b5266820ea16895399" Oct 30 02:21:33.345005 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:33.344976 5998 scope.go:122] "RemoveContainer" containerID="58e57e08c9d5dad7471ef7b0b26fe455e80e6e6a5286dc260d2ef3620a50d8e9" Oct 30 02:21:33.411961 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:33.411927 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:33.585149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:33.585048 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:33.836547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:33.836425 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:34.076115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:34.066376 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:34.332128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:34.330817 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:34.632638 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:34.632548 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:34.834073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:34.834036 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:35.979133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:35.979072 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:36.453811 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:36.453263 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:36.695897 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:36.695860 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:37.069241 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:37.069191 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:37.277363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:37.277281 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:37.780286 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:37.780248 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:37.949117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:37.940722 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:38.196616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:38.196581 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:38.348290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:38.348257 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:38.479618 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:38.479530 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:39.902940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:39.902901 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:40.048078 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:40.048043 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:40.297117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:40.296672 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:40.868111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:40.866729 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:41.511430 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:41.511387 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:41.713008 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:41.712973 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:41.840755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:41.840662 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:42.188114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:42.180973 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:42.677899 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:42.677861 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:42.922472 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:42.922429 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v"] Oct 30 02:21:42.923038 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:42.923021 5998 cpu_manager.go:414] "containerMap: removing container" podUID="428ef227-997d-4210-a1ef-de5cf92fc1a5" containerName="payload-processing" Oct 30 02:21:42.923083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:42.923040 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428ef227-997d-4210-a1ef-de5cf92fc1a5" containerName="payload-processing" Oct 30 02:21:42.923190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:42.923180 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="428ef227-997d-4210-a1ef-de5cf92fc1a5" containerName="payload-processing" Oct 30 02:21:42.928241 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:42.928196 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" Oct 30 02:21:42.950631 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:42.950590 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"kube-root-ca.crt\"" Oct 30 02:21:42.950811 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:42.950732 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"openshift-service-ca.crt\"" Oct 30 02:21:42.950880 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:42.950816 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"payload-processing-external-model-plugins-e2e-extmod-b-93ec46\"" Oct 30 02:21:42.961123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:42.961076 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"payload-processing-external-model-e2e--dd78327c-dockercfg-8hk6c\"" Oct 30 02:21:42.965753 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:42.965714 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e984db7b-b452-42a1-a514-b8f5655ef264-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v\" (UID: \"e984db7b-b452-42a1-a514-b8f5655ef264\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" Oct 30 02:21:42.965927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:42.965792 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e984db7b-b452-42a1-a514-b8f5655ef264-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v\" (UID: \"e984db7b-b452-42a1-a514-b8f5655ef264\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" Oct 30 02:21:42.965927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:42.965856 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e984db7b-b452-42a1-a514-b8f5655ef264-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v\" (UID: \"e984db7b-b452-42a1-a514-b8f5655ef264\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" Oct 30 02:21:42.966056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:42.965980 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29trz\" (UniqueName: \"kubernetes.io/projected/e984db7b-b452-42a1-a514-b8f5655ef264-kube-api-access-29trz\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v\" (UID: \"e984db7b-b452-42a1-a514-b8f5655ef264\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" Oct 30 02:21:42.966686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:42.966658 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v"] Oct 30 02:21:43.016107 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:43.016067 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"routing-overlay\"" Oct 30 02:21:43.046479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:43.046438 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"agc-cred-8974cf53f130\"" Oct 30 02:21:43.109184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:43.086602 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29trz\" (UniqueName: \"kubernetes.io/projected/e984db7b-b452-42a1-a514-b8f5655ef264-kube-api-access-29trz\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v\" (UID: \"e984db7b-b452-42a1-a514-b8f5655ef264\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" Oct 30 02:21:43.113283 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:43.109725 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e984db7b-b452-42a1-a514-b8f5655ef264-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v\" (UID: \"e984db7b-b452-42a1-a514-b8f5655ef264\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" Oct 30 02:21:43.113584 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:43.113558 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e984db7b-b452-42a1-a514-b8f5655ef264-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v\" (UID: \"e984db7b-b452-42a1-a514-b8f5655ef264\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" Oct 30 02:21:43.127282 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:43.127243 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e984db7b-b452-42a1-a514-b8f5655ef264-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v\" (UID: \"e984db7b-b452-42a1-a514-b8f5655ef264\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" Oct 30 02:21:43.127723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:43.113161 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e984db7b-b452-42a1-a514-b8f5655ef264-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v\" (UID: \"e984db7b-b452-42a1-a514-b8f5655ef264\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" Oct 30 02:21:43.127867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:43.114456 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e984db7b-b452-42a1-a514-b8f5655ef264-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v\" (UID: \"e984db7b-b452-42a1-a514-b8f5655ef264\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" Oct 30 02:21:43.153610 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:43.148909 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29trz\" (UniqueName: \"kubernetes.io/projected/e984db7b-b452-42a1-a514-b8f5655ef264-kube-api-access-29trz\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v\" (UID: \"e984db7b-b452-42a1-a514-b8f5655ef264\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" Oct 30 02:21:43.153610 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:43.152966 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e984db7b-b452-42a1-a514-b8f5655ef264-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v\" (UID: \"e984db7b-b452-42a1-a514-b8f5655ef264\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" Oct 30 02:21:43.157111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:43.154120 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:21:43.260945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:43.260853 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" Oct 30 02:21:43.454637 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:43.454547 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:43.698582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:43.697375 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v"] Oct 30 02:21:43.792116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:43.791512 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" event={"ID":"e984db7b-b452-42a1-a514-b8f5655ef264","Type":"ContainerStarted","Data":"a7d3f04b99b731789fae40104064d6c59b00442676b0d43b4690255bbb02a5bb"} Oct 30 02:21:44.805540 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:44.805498 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" event={"ID":"e984db7b-b452-42a1-a514-b8f5655ef264","Type":"ContainerStarted","Data":"af156524b47b80c2705fcf61a0c753c335c36007df395d7c5a796911bfabd177"} Oct 30 02:21:44.807330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:44.807304 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" Oct 30 02:21:44.850941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:44.850875 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" podStartSLOduration=2.850844837 podStartE2EDuration="2.850844837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:21:44.848653533 +0000 UTC m=+3134.427444614" watchObservedRunningTime="2026-10-30 02:21:44.850844837 +0000 UTC m=+3134.429635919" Oct 30 02:21:45.228012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:45.227976 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:45.619249 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:45.619157 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:45.768111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:45.759695 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:47.020287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:47.020232 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:21:47.095225 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:47.095164 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 02:21:47.098402 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:47.098368 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:21:47.215540 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:47.215495 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 02:21:47.932116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:47.917701 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq"] Oct 30 02:21:48.225273 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.185508 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 02:21:48.273215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.272238 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 02:21:48.307114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.301755 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 02:21:48.307114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.301815 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq"] Oct 30 02:21:48.307114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.302235 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" Oct 30 02:21:48.320015 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.319910 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-93ec46\"/\"agc-cred-67a916a223ea\"" Oct 30 02:21:48.344138 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.332687 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:48.474115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.458047 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr5q9\" (UniqueName: \"kubernetes.io/projected/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-kube-api-access-gr5q9\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq\" (UID: \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" Oct 30 02:21:48.474115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.458189 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq\" (UID: \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" Oct 30 02:21:48.474115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.458235 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq\" (UID: \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" Oct 30 02:21:48.474115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.458292 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq\" (UID: \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" Oct 30 02:21:48.600748 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.598552 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq\" (UID: \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" Oct 30 02:21:48.600748 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.598617 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq\" (UID: \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" Oct 30 02:21:48.600748 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.598662 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq\" (UID: \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" Oct 30 02:21:48.600748 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.598900 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr5q9\" (UniqueName: \"kubernetes.io/projected/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-kube-api-access-gr5q9\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq\" (UID: \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" Oct 30 02:21:48.600748 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.599874 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq\" (UID: \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" Oct 30 02:21:48.600748 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.600059 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq\" (UID: \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" Oct 30 02:21:48.628111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.625860 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr5q9\" (UniqueName: \"kubernetes.io/projected/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-kube-api-access-gr5q9\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq\" (UID: \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" Oct 30 02:21:48.634110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.629645 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 02:21:48.642185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.642081 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq\" (UID: \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\") " pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" Oct 30 02:21:48.653341 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:48.653069 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" Oct 30 02:21:49.004159 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:49.004126 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq"] Oct 30 02:21:49.005615 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:21:49.005579 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc698dd_2c33_4eaa_868d_9ebd6a706f4f.slice/crio-974e8c10685d4ef8346bef1d1eab0bef23bb97382336572a896051eeeaaea815 WatchSource:0}: Error finding container 974e8c10685d4ef8346bef1d1eab0bef23bb97382336572a896051eeeaaea815: Status 404 returned error can't find the container with id 974e8c10685d4ef8346bef1d1eab0bef23bb97382336572a896051eeeaaea815 Oct 30 02:21:49.261925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:49.261841 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" event={"ID":"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f","Type":"ContainerStarted","Data":"95e746b8299ff5ec0b8783aa4b1d7a3017dc91de1e4d9b37525335b78cc3881c"} Oct 30 02:21:49.261925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:49.261901 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" event={"ID":"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f","Type":"ContainerStarted","Data":"974e8c10685d4ef8346bef1d1eab0bef23bb97382336572a896051eeeaaea815"} Oct 30 02:21:49.262376 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:49.261949 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" Oct 30 02:21:49.282808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:49.282756 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:49.310036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:49.309971 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" podStartSLOduration=2.309953188 podStartE2EDuration="2.309953188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:21:49.306164594 +0000 UTC m=+3138.884955675" watchObservedRunningTime="2026-10-30 02:21:49.309953188 +0000 UTC m=+3138.888744272" Oct 30 02:21:49.446104 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:49.445975 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:50.100672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:50.100608 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:50.720163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:50.720127 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:50.892581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:50.892545 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:51.223398 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:51.223362 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:52.011014 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:52.010975 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:52.210161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:52.210126 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:52.400860 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:52.400765 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:52.563114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:52.563053 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:52.777847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:52.777808 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:53.099349 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:53.099273 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:53.250589 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:53.250552 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:53.445635 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:53.445577 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:53.601585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:53.601248 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:54.405618 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:54.405585 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:55.219215 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:21:55.219175 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:21:55.730427 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:55.730394 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:56.018532 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.018432 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr"] Oct 30 02:21:56.024043 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.024018 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" Oct 30 02:21:56.039328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.039288 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"agc-cred-6abdfa144479\"" Oct 30 02:21:56.048762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.048731 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"openshift-service-ca.crt\"" Oct 30 02:21:56.048987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.048792 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"payload-processing-external-model-e2e--c1fccfb3-dockercfg-fk79b\"" Oct 30 02:21:56.049295 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.049275 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"routing-overlay\"" Oct 30 02:21:56.049419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.049283 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"payload-processing-external-model-plugins-e2e-extmod-a-0ce950\"" Oct 30 02:21:56.049725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.049694 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-0ce950\"/\"kube-root-ca.crt\"" Oct 30 02:21:56.059121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.056914 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr"] Oct 30 02:21:56.105923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.105879 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2088c14-033f-4f6c-86a4-bf49540d3d67-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr\" (UID: \"f2088c14-033f-4f6c-86a4-bf49540d3d67\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" Oct 30 02:21:56.106169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.105938 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzm5b\" (UniqueName: \"kubernetes.io/projected/f2088c14-033f-4f6c-86a4-bf49540d3d67-kube-api-access-zzm5b\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr\" (UID: \"f2088c14-033f-4f6c-86a4-bf49540d3d67\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" Oct 30 02:21:56.106169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.106069 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2088c14-033f-4f6c-86a4-bf49540d3d67-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr\" (UID: \"f2088c14-033f-4f6c-86a4-bf49540d3d67\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" Oct 30 02:21:56.106340 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.106272 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2088c14-033f-4f6c-86a4-bf49540d3d67-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr\" (UID: \"f2088c14-033f-4f6c-86a4-bf49540d3d67\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" Oct 30 02:21:56.214672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.214627 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2088c14-033f-4f6c-86a4-bf49540d3d67-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr\" (UID: \"f2088c14-033f-4f6c-86a4-bf49540d3d67\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" Oct 30 02:21:56.244330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.244293 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2088c14-033f-4f6c-86a4-bf49540d3d67-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr\" (UID: \"f2088c14-033f-4f6c-86a4-bf49540d3d67\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" Oct 30 02:21:56.244570 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.244551 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzm5b\" (UniqueName: \"kubernetes.io/projected/f2088c14-033f-4f6c-86a4-bf49540d3d67-kube-api-access-zzm5b\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr\" (UID: \"f2088c14-033f-4f6c-86a4-bf49540d3d67\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" Oct 30 02:21:56.244861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.244841 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2088c14-033f-4f6c-86a4-bf49540d3d67-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr\" (UID: \"f2088c14-033f-4f6c-86a4-bf49540d3d67\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" Oct 30 02:21:56.262112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.255186 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2088c14-033f-4f6c-86a4-bf49540d3d67-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr\" (UID: \"f2088c14-033f-4f6c-86a4-bf49540d3d67\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" Oct 30 02:21:56.262112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.215827 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2088c14-033f-4f6c-86a4-bf49540d3d67-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr\" (UID: \"f2088c14-033f-4f6c-86a4-bf49540d3d67\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" Oct 30 02:21:56.262112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.258798 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2088c14-033f-4f6c-86a4-bf49540d3d67-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr\" (UID: \"f2088c14-033f-4f6c-86a4-bf49540d3d67\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" Oct 30 02:21:56.332173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.330011 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzm5b\" (UniqueName: \"kubernetes.io/projected/f2088c14-033f-4f6c-86a4-bf49540d3d67-kube-api-access-zzm5b\") pod \"payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr\" (UID: \"f2088c14-033f-4f6c-86a4-bf49540d3d67\") " pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" Oct 30 02:21:56.346138 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.342365 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" Oct 30 02:21:56.584406 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.584316 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:56.726624 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.726583 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr"] Oct 30 02:21:56.731665 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:21:56.731630 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2088c14_033f_4f6c_86a4_bf49540d3d67.slice/crio-7a944a7bdfe179d6545bf41b0c9c1737ddbff50dada9c80430064ce52060c5f6 WatchSource:0}: Error finding container 7a944a7bdfe179d6545bf41b0c9c1737ddbff50dada9c80430064ce52060c5f6: Status 404 returned error can't find the container with id 7a944a7bdfe179d6545bf41b0c9c1737ddbff50dada9c80430064ce52060c5f6 Oct 30 02:21:56.932248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:56.932209 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:57.106386 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:57.106351 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" Oct 30 02:21:57.429680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:57.428509 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" event={"ID":"f2088c14-033f-4f6c-86a4-bf49540d3d67","Type":"ContainerStarted","Data":"d8f076ce0bdf1e82c2c7f3ecedd180c975d76b81608ca68eef621b8ff25580bd"} Oct 30 02:21:57.429680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:57.428583 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" event={"ID":"f2088c14-033f-4f6c-86a4-bf49540d3d67","Type":"ContainerStarted","Data":"7a944a7bdfe179d6545bf41b0c9c1737ddbff50dada9c80430064ce52060c5f6"} Oct 30 02:21:57.429680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:57.429167 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" Oct 30 02:21:57.491113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:57.490910 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" podStartSLOduration=2.490888922 podStartE2EDuration="2.490888922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:21:57.488530263 +0000 UTC m=+3147.067321345" watchObservedRunningTime="2026-10-30 02:21:57.490888922 +0000 UTC m=+3147.069680003" Oct 30 02:21:58.492742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:58.492707 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:58.977864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:58.977826 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:59.136022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:59.135981 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:59.398829 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:59.398750 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:59.602164 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:59.602125 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:59.844179 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:59.844138 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:21:59.991544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:21:59.991511 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:00.476976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:00.476934 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:00.616980 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:00.616935 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:01.277984 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:01.277950 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" Oct 30 02:22:01.397083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:01.397050 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v"] Oct 30 02:22:01.847773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:01.846954 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:02.009170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:02.009127 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:02.221682 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:02.221222 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:03.065234 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.065191 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:03.208546 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.208508 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4"] Oct 30 02:22:03.213789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.213766 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" Oct 30 02:22:03.233603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.233572 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"agc-cred-20786862a813\"" Oct 30 02:22:03.233828 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.233650 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"openshift-service-ca.crt\"" Oct 30 02:22:03.233904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.233688 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"payload-processing-external-model-e2e--6aaede7f-dockercfg-j4qbl\"" Oct 30 02:22:03.250041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.250009 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4"] Oct 30 02:22:03.330467 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.330361 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-95f40f\"" Oct 30 02:22:03.330467 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.330375 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"routing-overlay\"" Oct 30 02:22:03.330467 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.330373 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"kube-root-ca.crt\"" Oct 30 02:22:03.333824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.333798 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-95f40f\"/\"agc-cred-43923dcebdf1\"" Oct 30 02:22:03.373486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.373446 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4\" (UID: \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" Oct 30 02:22:03.373679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.373605 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4\" (UID: \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" Oct 30 02:22:03.373679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.373670 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-596df\" (UniqueName: \"kubernetes.io/projected/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-kube-api-access-596df\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4\" (UID: \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" Oct 30 02:22:03.373838 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.373822 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4\" (UID: \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" Oct 30 02:22:03.475656 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.475613 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4\" (UID: \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" Oct 30 02:22:03.475882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.475697 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4\" (UID: \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" Oct 30 02:22:03.475882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.475766 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4\" (UID: \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" Oct 30 02:22:03.475882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.475807 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596df\" (UniqueName: \"kubernetes.io/projected/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-kube-api-access-596df\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4\" (UID: \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" Oct 30 02:22:03.476613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.476586 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4\" (UID: \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" Oct 30 02:22:03.476719 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.476614 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4\" (UID: \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" Oct 30 02:22:03.478164 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.478143 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4\" (UID: \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" Oct 30 02:22:03.499333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.499292 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-596df\" (UniqueName: \"kubernetes.io/projected/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-kube-api-access-596df\") pod \"payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4\" (UID: \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\") " pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" Oct 30 02:22:03.531251 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.531216 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" Oct 30 02:22:03.858737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:03.858704 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4"] Oct 30 02:22:03.861001 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:22:03.860961 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a70ffed_6e43_4b01_9bb8_7973c3c907e1.slice/crio-c284bbc29a522b26093505464025e591c26a5d5d25ec16fd1c8ce933f110cba8 WatchSource:0}: Error finding container c284bbc29a522b26093505464025e591c26a5d5d25ec16fd1c8ce933f110cba8: Status 404 returned error can't find the container with id c284bbc29a522b26093505464025e591c26a5d5d25ec16fd1c8ce933f110cba8 Oct 30 02:22:04.568372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:04.568335 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" event={"ID":"8a70ffed-6e43-4b01-9bb8-7973c3c907e1","Type":"ContainerStarted","Data":"539d12177b4a6c0c9ae8c0f3d9dd9621ec482dca3d7a36c51214df217dcce047"} Oct 30 02:22:04.568815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:04.568395 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" event={"ID":"8a70ffed-6e43-4b01-9bb8-7973c3c907e1","Type":"ContainerStarted","Data":"c284bbc29a522b26093505464025e591c26a5d5d25ec16fd1c8ce933f110cba8"} Oct 30 02:22:04.568815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:04.568435 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" Oct 30 02:22:04.614354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:04.614302 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" podStartSLOduration=1.614287792 podStartE2EDuration="1.614287792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:22:04.610294099 +0000 UTC m=+3154.189085178" watchObservedRunningTime="2026-10-30 02:22:04.614287792 +0000 UTC m=+3154.193078873" Oct 30 02:22:04.639741 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:04.639706 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:06.200987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:06.200954 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:06.398028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:06.397981 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" podUID="e984db7b-b452-42a1-a514-b8f5655ef264" containerName="payload-processing" containerID="cri-o://af156524b47b80c2705fcf61a0c753c335c36007df395d7c5a796911bfabd177" gracePeriod=25 Oct 30 02:22:06.594601 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:06.594331 5998 generic.go:356] "Generic (PLEG): container finished" podID="e984db7b-b452-42a1-a514-b8f5655ef264" containerID="af156524b47b80c2705fcf61a0c753c335c36007df395d7c5a796911bfabd177" exitCode=0 Oct 30 02:22:06.594601 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:06.594405 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" event={"ID":"e984db7b-b452-42a1-a514-b8f5655ef264","Type":"ContainerDied","Data":"af156524b47b80c2705fcf61a0c753c335c36007df395d7c5a796911bfabd177"} Oct 30 02:22:06.698150 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:06.698122 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" Oct 30 02:22:06.831552 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:06.831447 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e984db7b-b452-42a1-a514-b8f5655ef264-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e984db7b-b452-42a1-a514-b8f5655ef264-routing-overlay-volume\") pod \"e984db7b-b452-42a1-a514-b8f5655ef264\" (UID: \"e984db7b-b452-42a1-a514-b8f5655ef264\") " Oct 30 02:22:06.831808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:06.831576 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e984db7b-b452-42a1-a514-b8f5655ef264-kube-api-access-29trz\" (UniqueName: \"kubernetes.io/projected/e984db7b-b452-42a1-a514-b8f5655ef264-kube-api-access-29trz\") pod \"e984db7b-b452-42a1-a514-b8f5655ef264\" (UID: \"e984db7b-b452-42a1-a514-b8f5655ef264\") " Oct 30 02:22:06.831808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:06.831645 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e984db7b-b452-42a1-a514-b8f5655ef264-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e984db7b-b452-42a1-a514-b8f5655ef264-plugins-config-volume\") pod \"e984db7b-b452-42a1-a514-b8f5655ef264\" (UID: \"e984db7b-b452-42a1-a514-b8f5655ef264\") " Oct 30 02:22:06.831808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:06.831693 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e984db7b-b452-42a1-a514-b8f5655ef264-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e984db7b-b452-42a1-a514-b8f5655ef264-provider-credentials\") pod \"e984db7b-b452-42a1-a514-b8f5655ef264\" (UID: \"e984db7b-b452-42a1-a514-b8f5655ef264\") " Oct 30 02:22:06.832061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:06.832036 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e984db7b-b452-42a1-a514-b8f5655ef264-plugins-config-volume" pod "e984db7b-b452-42a1-a514-b8f5655ef264" (UID: "e984db7b-b452-42a1-a514-b8f5655ef264"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:06.832421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:06.832396 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e984db7b-b452-42a1-a514-b8f5655ef264-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:22:06.832623 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:06.832595 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e984db7b-b452-42a1-a514-b8f5655ef264-routing-overlay-volume" pod "e984db7b-b452-42a1-a514-b8f5655ef264" (UID: "e984db7b-b452-42a1-a514-b8f5655ef264"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:06.833969 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:06.833945 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e984db7b-b452-42a1-a514-b8f5655ef264-provider-credentials" pod "e984db7b-b452-42a1-a514-b8f5655ef264" (UID: "e984db7b-b452-42a1-a514-b8f5655ef264"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:06.834056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:06.834038 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e984db7b-b452-42a1-a514-b8f5655ef264-kube-api-access-29trz" pod "e984db7b-b452-42a1-a514-b8f5655ef264" (UID: "e984db7b-b452-42a1-a514-b8f5655ef264"). InnerVolumeSpecName "kube-api-access-29trz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:06.933343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:06.933310 5998 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e984db7b-b452-42a1-a514-b8f5655ef264-provider-credentials\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:22:06.933343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:06.933338 5998 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e984db7b-b452-42a1-a514-b8f5655ef264-routing-overlay-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:22:06.933343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:06.933348 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29trz\" (UniqueName: \"kubernetes.io/projected/e984db7b-b452-42a1-a514-b8f5655ef264-kube-api-access-29trz\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:22:07.603982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:07.603946 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" Oct 30 02:22:07.603982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:07.603955 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v" event={"ID":"e984db7b-b452-42a1-a514-b8f5655ef264","Type":"ContainerDied","Data":"a7d3f04b99b731789fae40104064d6c59b00442676b0d43b4690255bbb02a5bb"} Oct 30 02:22:07.604526 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:07.604032 5998 scope.go:122] "RemoveContainer" containerID="af156524b47b80c2705fcf61a0c753c335c36007df395d7c5a796911bfabd177" Oct 30 02:22:07.652487 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:07.652457 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v"] Oct 30 02:22:07.662654 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:07.662625 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-5c66wgp9v"] Oct 30 02:22:07.896164 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:07.896043 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:08.439990 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:08.439957 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" Oct 30 02:22:08.543330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:08.543280 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e984db7b-b452-42a1-a514-b8f5655ef264" path="/var/lib/kubelet/pods/e984db7b-b452-42a1-a514-b8f5655ef264/volumes" Oct 30 02:22:09.429904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:09.429858 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:11.107915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.107880 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:11.195041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.195001 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c"] Oct 30 02:22:11.196037 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.195939 5998 cpu_manager.go:414] "containerMap: removing container" podUID="e984db7b-b452-42a1-a514-b8f5655ef264" containerName="payload-processing" Oct 30 02:22:11.196037 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.195966 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e984db7b-b452-42a1-a514-b8f5655ef264" containerName="payload-processing" Oct 30 02:22:11.198132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.198103 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="e984db7b-b452-42a1-a514-b8f5655ef264" containerName="payload-processing" Oct 30 02:22:11.204567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.204536 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.225363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.225333 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 02:22:11.225363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.225335 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 02:22:11.225609 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.225417 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 02:22:11.230320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.230292 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c"] Oct 30 02:22:11.235770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.235743 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-v6t8d\"" Oct 30 02:22:11.305900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.305855 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r29g8\" (UniqueName: \"kubernetes.io/projected/21de7884-5d25-40b9-9cc4-5f9f87eff813-kube-api-access-r29g8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.306128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.305956 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.306128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.306063 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.306244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.306139 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.306294 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.306266 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.306347 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.306321 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.306461 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.306435 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21de7884-5d25-40b9-9cc4-5f9f87eff813-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.407715 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.407612 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.407715 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.407682 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.407976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.407748 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21de7884-5d25-40b9-9cc4-5f9f87eff813-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.407976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.407881 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r29g8\" (UniqueName: \"kubernetes.io/projected/21de7884-5d25-40b9-9cc4-5f9f87eff813-kube-api-access-r29g8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.407976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.407953 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.408181 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.408028 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.408181 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.408069 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.409024 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.408987 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.410390 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.410365 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21de7884-5d25-40b9-9cc4-5f9f87eff813-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.410931 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.410901 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.411032 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.410910 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.411467 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.411440 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.430042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.430011 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.430195 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.430125 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r29g8\" (UniqueName: \"kubernetes.io/projected/21de7884-5d25-40b9-9cc4-5f9f87eff813-kube-api-access-r29g8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-sc85c\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.521388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.521354 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:11.865390 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.865357 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c"] Oct 30 02:22:11.867480 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:22:11.867453 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21de7884_5d25_40b9_9cc4_5f9f87eff813.slice/crio-1500781dfc5f2096af685fb2fe7d68694ba5d1915e8de625de6b87a650e05ae7 WatchSource:0}: Error finding container 1500781dfc5f2096af685fb2fe7d68694ba5d1915e8de625de6b87a650e05ae7: Status 404 returned error can't find the container with id 1500781dfc5f2096af685fb2fe7d68694ba5d1915e8de625de6b87a650e05ae7 Oct 30 02:22:11.870559 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.870524 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:22:11.870653 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:11.870602 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:22:12.677059 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:12.677013 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" event={"ID":"21de7884-5d25-40b9-9cc4-5f9f87eff813","Type":"ContainerStarted","Data":"c420fbd1911f4930076a1c63a1bd3a9e1137fa4d4bdc0a515831d97f38732fcb"} Oct 30 02:22:12.677503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:12.677146 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" event={"ID":"21de7884-5d25-40b9-9cc4-5f9f87eff813","Type":"ContainerStarted","Data":"1500781dfc5f2096af685fb2fe7d68694ba5d1915e8de625de6b87a650e05ae7"} Oct 30 02:22:12.677686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:12.677664 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:12.679562 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:12.679531 5998 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-sc85c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:22:12.679713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:12.679584 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" podUID="21de7884-5d25-40b9-9cc4-5f9f87eff813" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:22:12.758839 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:12.756765 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" podStartSLOduration=1.756749542 podStartE2EDuration="1.756749542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:22:12.734775472 +0000 UTC m=+3162.313566554" watchObservedRunningTime="2026-10-30 02:22:12.756749542 +0000 UTC m=+3162.335540623" Oct 30 02:22:12.824738 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:12.824585 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:13.684831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:13.684798 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:22:14.575213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:14.575182 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:15.579808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:15.579777 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" Oct 30 02:22:16.205720 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:16.205685 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:17.999178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:17.999146 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:20.021158 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:20.017595 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:21.741987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:21.741948 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:22.436169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:22.436130 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt"] Oct 30 02:22:22.441588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:22.441563 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt" Oct 30 02:22:22.454490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:22.454463 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:22:22.465025 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:22.464970 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:22:22.465025 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:22.465017 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:22:22.465266 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:22.464980 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:22:22.471470 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:22.471441 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt"] Oct 30 02:22:22.598288 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:22.598253 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rd2b\" (UniqueName: \"kubernetes.io/projected/5018b551-1174-4f7c-b99f-b8c48154e68e-kube-api-access-2rd2b\") pod \"maas-api-revoke-sub-9ab7efce-zfnpt\" (UID: \"5018b551-1174-4f7c-b99f-b8c48154e68e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt" Oct 30 02:22:22.598494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:22.598309 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5018b551-1174-4f7c-b99f-b8c48154e68e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ab7efce-zfnpt\" (UID: \"5018b551-1174-4f7c-b99f-b8c48154e68e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt" Oct 30 02:22:22.598494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:22.598429 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5018b551-1174-4f7c-b99f-b8c48154e68e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ab7efce-zfnpt\" (UID: \"5018b551-1174-4f7c-b99f-b8c48154e68e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt" Oct 30 02:22:22.699684 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:22.699582 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rd2b\" (UniqueName: \"kubernetes.io/projected/5018b551-1174-4f7c-b99f-b8c48154e68e-kube-api-access-2rd2b\") pod \"maas-api-revoke-sub-9ab7efce-zfnpt\" (UID: \"5018b551-1174-4f7c-b99f-b8c48154e68e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt" Oct 30 02:22:22.699684 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:22.699679 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5018b551-1174-4f7c-b99f-b8c48154e68e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ab7efce-zfnpt\" (UID: \"5018b551-1174-4f7c-b99f-b8c48154e68e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt" Oct 30 02:22:22.699933 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:22.699801 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5018b551-1174-4f7c-b99f-b8c48154e68e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ab7efce-zfnpt\" (UID: \"5018b551-1174-4f7c-b99f-b8c48154e68e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt" Oct 30 02:22:22.700659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:22.700626 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5018b551-1174-4f7c-b99f-b8c48154e68e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ab7efce-zfnpt\" (UID: \"5018b551-1174-4f7c-b99f-b8c48154e68e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt" Oct 30 02:22:22.702161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:22.702137 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5018b551-1174-4f7c-b99f-b8c48154e68e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ab7efce-zfnpt\" (UID: \"5018b551-1174-4f7c-b99f-b8c48154e68e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt" Oct 30 02:22:22.763077 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:22.763047 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rd2b\" (UniqueName: \"kubernetes.io/projected/5018b551-1174-4f7c-b99f-b8c48154e68e-kube-api-access-2rd2b\") pod \"maas-api-revoke-sub-9ab7efce-zfnpt\" (UID: \"5018b551-1174-4f7c-b99f-b8c48154e68e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt" Oct 30 02:22:23.057723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:23.057637 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt" Oct 30 02:22:23.369942 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:23.369912 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt"] Oct 30 02:22:23.372148 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:22:23.372085 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5018b551_1174_4f7c_b99f_b8c48154e68e.slice/crio-f8342b44c61cce08732f8febd5d415f090a8528390fb2277f0eb1cda14c00ce3 WatchSource:0}: Error finding container f8342b44c61cce08732f8febd5d415f090a8528390fb2277f0eb1cda14c00ce3: Status 404 returned error can't find the container with id f8342b44c61cce08732f8febd5d415f090a8528390fb2277f0eb1cda14c00ce3 Oct 30 02:22:23.376546 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:23.376515 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:23.897940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:23.897838 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt" event={"ID":"5018b551-1174-4f7c-b99f-b8c48154e68e","Type":"ContainerStarted","Data":"9bb9f41b1fab6074d3fea1974e5ca8a6df5105db2613d722de6394440bc4fbfa"} Oct 30 02:22:23.897940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:23.897893 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt" event={"ID":"5018b551-1174-4f7c-b99f-b8c48154e68e","Type":"ContainerStarted","Data":"f8342b44c61cce08732f8febd5d415f090a8528390fb2277f0eb1cda14c00ce3"} Oct 30 02:22:23.936434 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:23.936378 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt" podStartSLOduration=1.936363795 podStartE2EDuration="1.936363795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:22:23.935274984 +0000 UTC m=+3173.514066065" watchObservedRunningTime="2026-10-30 02:22:23.936363795 +0000 UTC m=+3173.515154918" Oct 30 02:22:24.909371 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:24.909327 5998 generic.go:356] "Generic (PLEG): container finished" podID="5018b551-1174-4f7c-b99f-b8c48154e68e" containerID="9bb9f41b1fab6074d3fea1974e5ca8a6df5105db2613d722de6394440bc4fbfa" exitCode=0 Oct 30 02:22:24.909837 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:24.909406 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt" event={"ID":"5018b551-1174-4f7c-b99f-b8c48154e68e","Type":"ContainerDied","Data":"9bb9f41b1fab6074d3fea1974e5ca8a6df5105db2613d722de6394440bc4fbfa"} Oct 30 02:22:24.980509 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:24.980471 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:25.223840 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:22:25.223802 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:22:26.095363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:26.095333 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt" Oct 30 02:22:26.153980 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:26.153939 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5018b551-1174-4f7c-b99f-b8c48154e68e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5018b551-1174-4f7c-b99f-b8c48154e68e-maas-api-openshift-ca\") pod \"5018b551-1174-4f7c-b99f-b8c48154e68e\" (UID: \"5018b551-1174-4f7c-b99f-b8c48154e68e\") " Oct 30 02:22:26.154229 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:26.154111 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5018b551-1174-4f7c-b99f-b8c48154e68e-kube-api-access-2rd2b\" (UniqueName: \"kubernetes.io/projected/5018b551-1174-4f7c-b99f-b8c48154e68e-kube-api-access-2rd2b\") pod \"5018b551-1174-4f7c-b99f-b8c48154e68e\" (UID: \"5018b551-1174-4f7c-b99f-b8c48154e68e\") " Oct 30 02:22:26.154229 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:26.154215 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5018b551-1174-4f7c-b99f-b8c48154e68e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5018b551-1174-4f7c-b99f-b8c48154e68e-maas-api-xks-ca\") pod \"5018b551-1174-4f7c-b99f-b8c48154e68e\" (UID: \"5018b551-1174-4f7c-b99f-b8c48154e68e\") " Oct 30 02:22:26.154431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:26.154405 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5018b551-1174-4f7c-b99f-b8c48154e68e-maas-api-openshift-ca" pod "5018b551-1174-4f7c-b99f-b8c48154e68e" (UID: "5018b551-1174-4f7c-b99f-b8c48154e68e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:26.154894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:26.154868 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5018b551-1174-4f7c-b99f-b8c48154e68e-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:22:26.166120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:26.156526 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5018b551-1174-4f7c-b99f-b8c48154e68e-kube-api-access-2rd2b" pod "5018b551-1174-4f7c-b99f-b8c48154e68e" (UID: "5018b551-1174-4f7c-b99f-b8c48154e68e"). InnerVolumeSpecName "kube-api-access-2rd2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:26.166120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:26.156790 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5018b551-1174-4f7c-b99f-b8c48154e68e-maas-api-xks-ca" pod "5018b551-1174-4f7c-b99f-b8c48154e68e" (UID: "5018b551-1174-4f7c-b99f-b8c48154e68e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:22:26.255843 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:26.255807 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rd2b\" (UniqueName: \"kubernetes.io/projected/5018b551-1174-4f7c-b99f-b8c48154e68e-kube-api-access-2rd2b\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:22:26.255843 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:26.255837 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5018b551-1174-4f7c-b99f-b8c48154e68e-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:22:26.584372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:26.584312 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:26.929808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:26.929778 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt" Oct 30 02:22:26.930006 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:26.929827 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt" event={"ID":"5018b551-1174-4f7c-b99f-b8c48154e68e","Type":"ContainerDied","Data":"f8342b44c61cce08732f8febd5d415f090a8528390fb2277f0eb1cda14c00ce3"} Oct 30 02:22:26.930006 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:26.929906 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8342b44c61cce08732f8febd5d415f090a8528390fb2277f0eb1cda14c00ce3" Oct 30 02:22:28.034526 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:28.034487 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g"] Oct 30 02:22:28.046870 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:28.046833 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27be6abd-sbf6g"] Oct 30 02:22:28.285964 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:28.285876 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:28.539986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:28.539883 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb8ba250-629d-4170-8521-c4909c336dfb" path="/var/lib/kubelet/pods/fb8ba250-629d-4170-8521-c4909c336dfb/volumes" Oct 30 02:22:29.897756 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:29.897710 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:31.488074 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:31.488036 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:33.245553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:33.245517 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:33.780243 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:33.780205 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:33.908805 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:33.908768 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:34.025253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:34.025211 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:34.135380 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:34.135291 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:34.927064 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:34.927022 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:35.080109 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:35.080058 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:35.223441 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:35.223321 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:35.591850 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:35.591767 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:35.718048 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:35.718013 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:37.203178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:37.203132 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:38.493120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:38.493055 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:38.817470 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:38.817385 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:40.448600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:40.448493 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:43.610445 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:43.610398 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:43.762174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:43.762135 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:43.905917 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:43.905827 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:45.427395 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:45.427351 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:47.209752 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:47.209710 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:48.032133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:48.032065 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh"] Oct 30 02:22:48.042312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:48.042280 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-498006ff-xjwnh"] Oct 30 02:22:48.544403 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:48.544361 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b9afa50-aa53-483b-96fd-327c6f0055ab" path="/var/lib/kubelet/pods/6b9afa50-aa53-483b-96fd-327c6f0055ab/volumes" Oct 30 02:22:48.871510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:48.871396 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:49.953244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:49.953209 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:50.061673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:50.061627 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:50.508247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:50.508211 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:50.884214 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:50.884113 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:51.043492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:51.043452 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:51.832342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:51.832303 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:22:52.250994 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:52.250953 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:52.379647 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:52.379605 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:54.106803 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:54.106760 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:55.219678 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:22:55.219637 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:22:55.605394 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.605313 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q"] Oct 30 02:22:55.606065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.606040 5998 cpu_manager.go:414] "containerMap: removing container" podUID="5018b551-1174-4f7c-b99f-b8c48154e68e" containerName="revoke-keys" Oct 30 02:22:55.606065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.606066 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5018b551-1174-4f7c-b99f-b8c48154e68e" containerName="revoke-keys" Oct 30 02:22:55.606298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.606250 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="5018b551-1174-4f7c-b99f-b8c48154e68e" containerName="revoke-keys" Oct 30 02:22:55.611241 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.611220 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q" Oct 30 02:22:55.624163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.624141 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:22:55.624882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.624857 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:55.634276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.634232 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:22:55.634276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.634271 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:22:55.634488 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.634331 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:22:55.640896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.640868 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q"] Oct 30 02:22:55.700963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.699062 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6142a278-891e-4516-820c-4e6c3b29fac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b4faac4-jzn4q\" (UID: \"6142a278-891e-4516-820c-4e6c3b29fac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q" Oct 30 02:22:55.700963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.699210 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6142a278-891e-4516-820c-4e6c3b29fac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b4faac4-jzn4q\" (UID: \"6142a278-891e-4516-820c-4e6c3b29fac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q" Oct 30 02:22:55.700963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.699269 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtnqt\" (UniqueName: \"kubernetes.io/projected/6142a278-891e-4516-820c-4e6c3b29fac0-kube-api-access-qtnqt\") pod \"maas-api-revoke-sub-8b4faac4-jzn4q\" (UID: \"6142a278-891e-4516-820c-4e6c3b29fac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q" Oct 30 02:22:55.801433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.801396 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6142a278-891e-4516-820c-4e6c3b29fac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b4faac4-jzn4q\" (UID: \"6142a278-891e-4516-820c-4e6c3b29fac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q" Oct 30 02:22:55.801646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.801505 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6142a278-891e-4516-820c-4e6c3b29fac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b4faac4-jzn4q\" (UID: \"6142a278-891e-4516-820c-4e6c3b29fac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q" Oct 30 02:22:55.801646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.801553 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtnqt\" (UniqueName: \"kubernetes.io/projected/6142a278-891e-4516-820c-4e6c3b29fac0-kube-api-access-qtnqt\") pod \"maas-api-revoke-sub-8b4faac4-jzn4q\" (UID: \"6142a278-891e-4516-820c-4e6c3b29fac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q" Oct 30 02:22:55.802315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.802291 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6142a278-891e-4516-820c-4e6c3b29fac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b4faac4-jzn4q\" (UID: \"6142a278-891e-4516-820c-4e6c3b29fac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q" Oct 30 02:22:55.803933 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.803908 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6142a278-891e-4516-820c-4e6c3b29fac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b4faac4-jzn4q\" (UID: \"6142a278-891e-4516-820c-4e6c3b29fac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q" Oct 30 02:22:55.822531 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.822504 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtnqt\" (UniqueName: \"kubernetes.io/projected/6142a278-891e-4516-820c-4e6c3b29fac0-kube-api-access-qtnqt\") pod \"maas-api-revoke-sub-8b4faac4-jzn4q\" (UID: \"6142a278-891e-4516-820c-4e6c3b29fac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q" Oct 30 02:22:55.926965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:55.926931 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q" Oct 30 02:22:56.234914 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:56.234799 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q"] Oct 30 02:22:56.240242 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:22:56.240211 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6142a278_891e_4516_820c_4e6c3b29fac0.slice/crio-55c758abd4097b2388cf23b1014d526e7c228c06344c0a9f0e1d67dea0a374de WatchSource:0}: Error finding container 55c758abd4097b2388cf23b1014d526e7c228c06344c0a9f0e1d67dea0a374de: Status 404 returned error can't find the container with id 55c758abd4097b2388cf23b1014d526e7c228c06344c0a9f0e1d67dea0a374de Oct 30 02:22:56.354491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:56.354455 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q" event={"ID":"6142a278-891e-4516-820c-4e6c3b29fac0","Type":"ContainerStarted","Data":"55c758abd4097b2388cf23b1014d526e7c228c06344c0a9f0e1d67dea0a374de"} Oct 30 02:22:57.300968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:57.300935 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:57.387121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:57.380368 5998 generic.go:356] "Generic (PLEG): container finished" podID="6142a278-891e-4516-820c-4e6c3b29fac0" containerID="13a8035afaff1aba96ff537059f0b0f2d24242ec91e791fda052bfa10d0d5e5d" exitCode=0 Oct 30 02:22:57.387121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:57.380536 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q" event={"ID":"6142a278-891e-4516-820c-4e6c3b29fac0","Type":"ContainerDied","Data":"13a8035afaff1aba96ff537059f0b0f2d24242ec91e791fda052bfa10d0d5e5d"} Oct 30 02:22:58.581644 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:58.581616 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q" Oct 30 02:22:58.746319 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:58.746275 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6142a278-891e-4516-820c-4e6c3b29fac0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6142a278-891e-4516-820c-4e6c3b29fac0-maas-api-openshift-ca\") pod \"6142a278-891e-4516-820c-4e6c3b29fac0\" (UID: \"6142a278-891e-4516-820c-4e6c3b29fac0\") " Oct 30 02:22:58.746548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:58.746329 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6142a278-891e-4516-820c-4e6c3b29fac0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6142a278-891e-4516-820c-4e6c3b29fac0-maas-api-xks-ca\") pod \"6142a278-891e-4516-820c-4e6c3b29fac0\" (UID: \"6142a278-891e-4516-820c-4e6c3b29fac0\") " Oct 30 02:22:58.746548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:58.746539 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6142a278-891e-4516-820c-4e6c3b29fac0-kube-api-access-qtnqt\" (UniqueName: \"kubernetes.io/projected/6142a278-891e-4516-820c-4e6c3b29fac0-kube-api-access-qtnqt\") pod \"6142a278-891e-4516-820c-4e6c3b29fac0\" (UID: \"6142a278-891e-4516-820c-4e6c3b29fac0\") " Oct 30 02:22:58.746756 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:58.746729 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6142a278-891e-4516-820c-4e6c3b29fac0-maas-api-openshift-ca" pod "6142a278-891e-4516-820c-4e6c3b29fac0" (UID: "6142a278-891e-4516-820c-4e6c3b29fac0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:58.747183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:58.747158 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6142a278-891e-4516-820c-4e6c3b29fac0-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:22:58.748523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:58.748499 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6142a278-891e-4516-820c-4e6c3b29fac0-maas-api-xks-ca" pod "6142a278-891e-4516-820c-4e6c3b29fac0" (UID: "6142a278-891e-4516-820c-4e6c3b29fac0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:22:58.748622 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:58.748605 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6142a278-891e-4516-820c-4e6c3b29fac0-kube-api-access-qtnqt" pod "6142a278-891e-4516-820c-4e6c3b29fac0" (UID: "6142a278-891e-4516-820c-4e6c3b29fac0"). InnerVolumeSpecName "kube-api-access-qtnqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:58.848736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:58.848699 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtnqt\" (UniqueName: \"kubernetes.io/projected/6142a278-891e-4516-820c-4e6c3b29fac0-kube-api-access-qtnqt\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:22:58.848736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:58.848730 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6142a278-891e-4516-820c-4e6c3b29fac0-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:22:58.978123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:58.978076 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:22:59.398562 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:59.398535 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q" Oct 30 02:22:59.398745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:59.398560 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q" event={"ID":"6142a278-891e-4516-820c-4e6c3b29fac0","Type":"ContainerDied","Data":"55c758abd4097b2388cf23b1014d526e7c228c06344c0a9f0e1d67dea0a374de"} Oct 30 02:22:59.398745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:22:59.398630 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55c758abd4097b2388cf23b1014d526e7c228c06344c0a9f0e1d67dea0a374de" Oct 30 02:23:00.673934 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:00.673895 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:02.217013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:02.216981 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:03.788763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:03.788717 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:05.388176 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:05.388137 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:06.258360 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.258320 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq"] Oct 30 02:23:06.258964 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.258949 5998 cpu_manager.go:414] "containerMap: removing container" podUID="6142a278-891e-4516-820c-4e6c3b29fac0" containerName="revoke-keys" Oct 30 02:23:06.259018 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.258965 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6142a278-891e-4516-820c-4e6c3b29fac0" containerName="revoke-keys" Oct 30 02:23:06.259063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.259054 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="6142a278-891e-4516-820c-4e6c3b29fac0" containerName="revoke-keys" Oct 30 02:23:06.262553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.262528 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq" Oct 30 02:23:06.283755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.283721 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq"] Oct 30 02:23:06.288903 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.288870 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:23:06.289044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.288932 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:23:06.289044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.288948 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:23:06.289167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.289072 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:23:06.367682 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.367643 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a66c194f-7609-49ca-9fcc-81285e0b60c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a94b656-xbqqq\" (UID: \"a66c194f-7609-49ca-9fcc-81285e0b60c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq" Oct 30 02:23:06.367876 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.367700 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a66c194f-7609-49ca-9fcc-81285e0b60c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a94b656-xbqqq\" (UID: \"a66c194f-7609-49ca-9fcc-81285e0b60c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq" Oct 30 02:23:06.367876 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.367797 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzgl8\" (UniqueName: \"kubernetes.io/projected/a66c194f-7609-49ca-9fcc-81285e0b60c8-kube-api-access-wzgl8\") pod \"maas-api-revoke-sub-0a94b656-xbqqq\" (UID: \"a66c194f-7609-49ca-9fcc-81285e0b60c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq" Oct 30 02:23:06.469041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.469003 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a66c194f-7609-49ca-9fcc-81285e0b60c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a94b656-xbqqq\" (UID: \"a66c194f-7609-49ca-9fcc-81285e0b60c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq" Oct 30 02:23:06.469500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.469055 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a66c194f-7609-49ca-9fcc-81285e0b60c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a94b656-xbqqq\" (UID: \"a66c194f-7609-49ca-9fcc-81285e0b60c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq" Oct 30 02:23:06.469500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.469133 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzgl8\" (UniqueName: \"kubernetes.io/projected/a66c194f-7609-49ca-9fcc-81285e0b60c8-kube-api-access-wzgl8\") pod \"maas-api-revoke-sub-0a94b656-xbqqq\" (UID: \"a66c194f-7609-49ca-9fcc-81285e0b60c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq" Oct 30 02:23:06.469892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.469863 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a66c194f-7609-49ca-9fcc-81285e0b60c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a94b656-xbqqq\" (UID: \"a66c194f-7609-49ca-9fcc-81285e0b60c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq" Oct 30 02:23:06.471632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.471602 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a66c194f-7609-49ca-9fcc-81285e0b60c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a94b656-xbqqq\" (UID: \"a66c194f-7609-49ca-9fcc-81285e0b60c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq" Oct 30 02:23:06.491266 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.491238 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzgl8\" (UniqueName: \"kubernetes.io/projected/a66c194f-7609-49ca-9fcc-81285e0b60c8-kube-api-access-wzgl8\") pod \"maas-api-revoke-sub-0a94b656-xbqqq\" (UID: \"a66c194f-7609-49ca-9fcc-81285e0b60c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq" Oct 30 02:23:06.576295 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.576215 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq" Oct 30 02:23:06.892220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:06.892194 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq"] Oct 30 02:23:06.894734 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:23:06.894693 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66c194f_7609_49ca_9fcc_81285e0b60c8.slice/crio-d2b927dc124dab3ea618cd3f1593d5b632e0cb2e2b406bf82cf0116ae3d2740f WatchSource:0}: Error finding container d2b927dc124dab3ea618cd3f1593d5b632e0cb2e2b406bf82cf0116ae3d2740f: Status 404 returned error can't find the container with id d2b927dc124dab3ea618cd3f1593d5b632e0cb2e2b406bf82cf0116ae3d2740f Oct 30 02:23:07.081470 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:07.081437 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:07.489067 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:07.488965 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq" event={"ID":"a66c194f-7609-49ca-9fcc-81285e0b60c8","Type":"ContainerStarted","Data":"fb919ea9b020bb5e950dfcdac79b7548ac6f84c624ca9dce0fa45327304132bd"} Oct 30 02:23:07.489452 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:07.489074 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq" event={"ID":"a66c194f-7609-49ca-9fcc-81285e0b60c8","Type":"ContainerStarted","Data":"d2b927dc124dab3ea618cd3f1593d5b632e0cb2e2b406bf82cf0116ae3d2740f"} Oct 30 02:23:07.529595 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:07.529539 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq" podStartSLOduration=1.5295250660000002 podStartE2EDuration="1.529525066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:23:07.525462581 +0000 UTC m=+3217.104253661" watchObservedRunningTime="2026-10-30 02:23:07.529525066 +0000 UTC m=+3217.108316147" Oct 30 02:23:08.498884 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:08.498842 5998 generic.go:356] "Generic (PLEG): container finished" podID="a66c194f-7609-49ca-9fcc-81285e0b60c8" containerID="fb919ea9b020bb5e950dfcdac79b7548ac6f84c624ca9dce0fa45327304132bd" exitCode=0 Oct 30 02:23:08.499425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:08.498927 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq" event={"ID":"a66c194f-7609-49ca-9fcc-81285e0b60c8","Type":"ContainerDied","Data":"fb919ea9b020bb5e950dfcdac79b7548ac6f84c624ca9dce0fa45327304132bd"} Oct 30 02:23:08.688891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:08.688859 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:09.685009 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:09.684979 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq" Oct 30 02:23:09.727342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:09.725146 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a66c194f-7609-49ca-9fcc-81285e0b60c8-kube-api-access-wzgl8\" (UniqueName: \"kubernetes.io/projected/a66c194f-7609-49ca-9fcc-81285e0b60c8-kube-api-access-wzgl8\") pod \"a66c194f-7609-49ca-9fcc-81285e0b60c8\" (UID: \"a66c194f-7609-49ca-9fcc-81285e0b60c8\") " Oct 30 02:23:09.727342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:09.725317 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a66c194f-7609-49ca-9fcc-81285e0b60c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a66c194f-7609-49ca-9fcc-81285e0b60c8-maas-api-openshift-ca\") pod \"a66c194f-7609-49ca-9fcc-81285e0b60c8\" (UID: \"a66c194f-7609-49ca-9fcc-81285e0b60c8\") " Oct 30 02:23:09.727342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:09.725470 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a66c194f-7609-49ca-9fcc-81285e0b60c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a66c194f-7609-49ca-9fcc-81285e0b60c8-maas-api-xks-ca\") pod \"a66c194f-7609-49ca-9fcc-81285e0b60c8\" (UID: \"a66c194f-7609-49ca-9fcc-81285e0b60c8\") " Oct 30 02:23:09.727342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:09.725996 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a66c194f-7609-49ca-9fcc-81285e0b60c8-maas-api-openshift-ca" pod "a66c194f-7609-49ca-9fcc-81285e0b60c8" (UID: "a66c194f-7609-49ca-9fcc-81285e0b60c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:23:09.727342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:09.726354 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a66c194f-7609-49ca-9fcc-81285e0b60c8-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:23:09.731603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:09.731576 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a66c194f-7609-49ca-9fcc-81285e0b60c8-maas-api-xks-ca" pod "a66c194f-7609-49ca-9fcc-81285e0b60c8" (UID: "a66c194f-7609-49ca-9fcc-81285e0b60c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:23:09.731702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:09.731641 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a66c194f-7609-49ca-9fcc-81285e0b60c8-kube-api-access-wzgl8" pod "a66c194f-7609-49ca-9fcc-81285e0b60c8" (UID: "a66c194f-7609-49ca-9fcc-81285e0b60c8"). InnerVolumeSpecName "kube-api-access-wzgl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:23:09.828412 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:09.828320 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a66c194f-7609-49ca-9fcc-81285e0b60c8-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:23:09.828412 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:09.828355 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzgl8\" (UniqueName: \"kubernetes.io/projected/a66c194f-7609-49ca-9fcc-81285e0b60c8-kube-api-access-wzgl8\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:23:10.293951 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:10.293921 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:10.520856 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:10.520817 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq" Oct 30 02:23:10.521059 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:10.520844 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq" event={"ID":"a66c194f-7609-49ca-9fcc-81285e0b60c8","Type":"ContainerDied","Data":"d2b927dc124dab3ea618cd3f1593d5b632e0cb2e2b406bf82cf0116ae3d2740f"} Oct 30 02:23:10.521059 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:10.520909 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2b927dc124dab3ea618cd3f1593d5b632e0cb2e2b406bf82cf0116ae3d2740f" Oct 30 02:23:11.990764 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:11.990729 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:13.500700 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:13.500666 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:15.187391 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:15.187353 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:16.880516 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:16.880449 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:17.609124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:17.602076 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:17.842113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:17.824943 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:18.843980 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:18.842076 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:18.962761 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:18.957227 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:19.038271 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:19.038225 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 02:23:19.085269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:19.085217 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:23:19.099615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:19.086865 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:19.109126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:19.100985 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:23:19.247227 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:19.247177 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 02:23:19.332400 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:19.257183 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 02:23:19.333180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:19.257219 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 02:23:19.395348 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:19.395176 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:19.443459 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:19.442197 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:23:19.551125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:19.548303 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:20.051233 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.051179 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:23:20.052664 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.052638 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 02:23:20.088213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.088168 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 02:23:20.089352 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.089324 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 02:23:20.113323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.113275 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 02:23:20.113971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.113943 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 02:23:20.132473 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.132432 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 02:23:20.208358 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.207289 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-57s4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:23:20.208358 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.207377 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" podUID="caaec13d-e62d-4450-864c-d9e3fb7ca8d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:23:20.208358 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.208291 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 02:23:20.336226 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.323823 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:23:20.352142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.336530 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:23:20.395498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.395465 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 02:23:20.395772 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.395759 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 02:23:20.395890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.395880 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 02:23:20.435584 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.435550 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 02:23:20.843206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.843164 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6"] Oct 30 02:23:20.845053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.845002 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a66c194f-7609-49ca-9fcc-81285e0b60c8" containerName="revoke-keys" Oct 30 02:23:20.845265 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.845247 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a66c194f-7609-49ca-9fcc-81285e0b60c8" containerName="revoke-keys" Oct 30 02:23:20.845727 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.845710 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a66c194f-7609-49ca-9fcc-81285e0b60c8" containerName="revoke-keys" Oct 30 02:23:20.853174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.852150 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6" Oct 30 02:23:20.858924 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.852016 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6"] Oct 30 02:23:20.866912 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.866883 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:23:20.870073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.870038 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:23:20.874416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.870440 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:23:20.874622 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.870533 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:23:20.919819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.919771 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032e9f87-821b-4408-9407-0ed3ce36dbfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4e747a6-zcjd6\" (UID: \"032e9f87-821b-4408-9407-0ed3ce36dbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6" Oct 30 02:23:20.920047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.920026 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032e9f87-821b-4408-9407-0ed3ce36dbfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4e747a6-zcjd6\" (UID: \"032e9f87-821b-4408-9407-0ed3ce36dbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6" Oct 30 02:23:20.920265 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:20.920242 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8gns\" (UniqueName: \"kubernetes.io/projected/032e9f87-821b-4408-9407-0ed3ce36dbfb-kube-api-access-c8gns\") pod \"maas-api-revoke-sub-f4e747a6-zcjd6\" (UID: \"032e9f87-821b-4408-9407-0ed3ce36dbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6" Oct 30 02:23:21.021354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:21.021319 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032e9f87-821b-4408-9407-0ed3ce36dbfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4e747a6-zcjd6\" (UID: \"032e9f87-821b-4408-9407-0ed3ce36dbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6" Oct 30 02:23:21.021686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:21.021444 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032e9f87-821b-4408-9407-0ed3ce36dbfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4e747a6-zcjd6\" (UID: \"032e9f87-821b-4408-9407-0ed3ce36dbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6" Oct 30 02:23:21.021686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:21.021503 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8gns\" (UniqueName: \"kubernetes.io/projected/032e9f87-821b-4408-9407-0ed3ce36dbfb-kube-api-access-c8gns\") pod \"maas-api-revoke-sub-f4e747a6-zcjd6\" (UID: \"032e9f87-821b-4408-9407-0ed3ce36dbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6" Oct 30 02:23:21.022162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:21.022134 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032e9f87-821b-4408-9407-0ed3ce36dbfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4e747a6-zcjd6\" (UID: \"032e9f87-821b-4408-9407-0ed3ce36dbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6" Oct 30 02:23:21.024513 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:21.024486 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032e9f87-821b-4408-9407-0ed3ce36dbfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4e747a6-zcjd6\" (UID: \"032e9f87-821b-4408-9407-0ed3ce36dbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6" Oct 30 02:23:21.070420 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:21.070377 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8gns\" (UniqueName: \"kubernetes.io/projected/032e9f87-821b-4408-9407-0ed3ce36dbfb-kube-api-access-c8gns\") pod \"maas-api-revoke-sub-f4e747a6-zcjd6\" (UID: \"032e9f87-821b-4408-9407-0ed3ce36dbfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6" Oct 30 02:23:21.180298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:21.180262 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6" Oct 30 02:23:21.367787 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:21.367753 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:21.517729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:21.517697 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6"] Oct 30 02:23:21.521491 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:23:21.521454 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032e9f87_821b_4408_9407_0ed3ce36dbfb.slice/crio-f0e51bb445ce07f4affc27dddbc1bdb084aa4e483fcba7879c2ed618a6fdbbb2 WatchSource:0}: Error finding container f0e51bb445ce07f4affc27dddbc1bdb084aa4e483fcba7879c2ed618a6fdbbb2: Status 404 returned error can't find the container with id f0e51bb445ce07f4affc27dddbc1bdb084aa4e483fcba7879c2ed618a6fdbbb2 Oct 30 02:23:22.027729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:22.027690 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6" event={"ID":"032e9f87-821b-4408-9407-0ed3ce36dbfb","Type":"ContainerStarted","Data":"9b89f8190f820871e320dacda823224fef4cb21d353d0af9d6544350e0773f16"} Oct 30 02:23:22.027908 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:22.027758 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6" event={"ID":"032e9f87-821b-4408-9407-0ed3ce36dbfb","Type":"ContainerStarted","Data":"f0e51bb445ce07f4affc27dddbc1bdb084aa4e483fcba7879c2ed618a6fdbbb2"} Oct 30 02:23:22.068772 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:22.068663 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6" podStartSLOduration=2.068650178 podStartE2EDuration="2.068650178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:23:22.064834391 +0000 UTC m=+3231.643625472" watchObservedRunningTime="2026-10-30 02:23:22.068650178 +0000 UTC m=+3231.647441259" Oct 30 02:23:22.963607 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:22.963541 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:23.055458 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:23.055419 5998 generic.go:356] "Generic (PLEG): container finished" podID="032e9f87-821b-4408-9407-0ed3ce36dbfb" containerID="9b89f8190f820871e320dacda823224fef4cb21d353d0af9d6544350e0773f16" exitCode=0 Oct 30 02:23:23.055654 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:23.055602 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6" event={"ID":"032e9f87-821b-4408-9407-0ed3ce36dbfb","Type":"ContainerDied","Data":"9b89f8190f820871e320dacda823224fef4cb21d353d0af9d6544350e0773f16"} Oct 30 02:23:24.245838 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:24.245805 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6" Oct 30 02:23:24.387705 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:24.387618 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/032e9f87-821b-4408-9407-0ed3ce36dbfb-kube-api-access-c8gns\" (UniqueName: \"kubernetes.io/projected/032e9f87-821b-4408-9407-0ed3ce36dbfb-kube-api-access-c8gns\") pod \"032e9f87-821b-4408-9407-0ed3ce36dbfb\" (UID: \"032e9f87-821b-4408-9407-0ed3ce36dbfb\") " Oct 30 02:23:24.387875 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:24.387705 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/032e9f87-821b-4408-9407-0ed3ce36dbfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032e9f87-821b-4408-9407-0ed3ce36dbfb-maas-api-xks-ca\") pod \"032e9f87-821b-4408-9407-0ed3ce36dbfb\" (UID: \"032e9f87-821b-4408-9407-0ed3ce36dbfb\") " Oct 30 02:23:24.387875 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:24.387760 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/032e9f87-821b-4408-9407-0ed3ce36dbfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032e9f87-821b-4408-9407-0ed3ce36dbfb-maas-api-openshift-ca\") pod \"032e9f87-821b-4408-9407-0ed3ce36dbfb\" (UID: \"032e9f87-821b-4408-9407-0ed3ce36dbfb\") " Oct 30 02:23:24.388222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:24.388196 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032e9f87-821b-4408-9407-0ed3ce36dbfb-maas-api-openshift-ca" pod "032e9f87-821b-4408-9407-0ed3ce36dbfb" (UID: "032e9f87-821b-4408-9407-0ed3ce36dbfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:23:24.388539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:24.388516 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/032e9f87-821b-4408-9407-0ed3ce36dbfb-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:23:24.389812 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:24.389784 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/032e9f87-821b-4408-9407-0ed3ce36dbfb-maas-api-xks-ca" pod "032e9f87-821b-4408-9407-0ed3ce36dbfb" (UID: "032e9f87-821b-4408-9407-0ed3ce36dbfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:23:24.390057 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:24.390035 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032e9f87-821b-4408-9407-0ed3ce36dbfb-kube-api-access-c8gns" pod "032e9f87-821b-4408-9407-0ed3ce36dbfb" (UID: "032e9f87-821b-4408-9407-0ed3ce36dbfb"). InnerVolumeSpecName "kube-api-access-c8gns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:23:24.489546 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:24.489512 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8gns\" (UniqueName: \"kubernetes.io/projected/032e9f87-821b-4408-9407-0ed3ce36dbfb-kube-api-access-c8gns\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:23:24.489546 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:24.489544 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/032e9f87-821b-4408-9407-0ed3ce36dbfb-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:23:24.566802 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:24.566769 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:25.078463 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:25.078428 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6" Oct 30 02:23:25.078653 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:25.078432 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6" event={"ID":"032e9f87-821b-4408-9407-0ed3ce36dbfb","Type":"ContainerDied","Data":"f0e51bb445ce07f4affc27dddbc1bdb084aa4e483fcba7879c2ed618a6fdbbb2"} Oct 30 02:23:25.078653 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:25.078582 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0e51bb445ce07f4affc27dddbc1bdb084aa4e483fcba7879c2ed618a6fdbbb2" Oct 30 02:23:25.225765 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:23:25.225716 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:23:26.079534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:26.079497 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:26.190654 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:26.190620 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:26.306269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:26.306213 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:27.770767 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:27.770716 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:29.350416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:29.350383 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:31.119553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:31.119513 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:32.415442 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:32.415404 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:32.552114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:32.543368 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:33.264294 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:33.264255 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:33.425505 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:33.425470 5998 scope.go:122] "RemoveContainer" containerID="b6525ea88038fc379ec0c73f962ce07280e2b4779fc8bd8b1fd209dbf390fd7c" Oct 30 02:23:33.558185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:33.558033 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:33.670914 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:33.670878 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:34.479287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:34.479158 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:34.611832 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:34.611794 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:35.143279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:35.143232 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:35.251864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:35.251832 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:35.426436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:35.426388 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:37.051060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:37.050993 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:38.614627 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:38.614594 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:38.798129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:38.798056 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:38.936829 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:38.936790 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:39.085749 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:39.085713 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:40.612079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:40.612040 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:40.773837 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:40.773798 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:42.359118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:42.359022 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:43.977996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:43.977958 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:45.645459 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:45.645394 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:47.246962 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:47.246924 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:48.786353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:48.786315 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:50.357813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:50.357769 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:51.874341 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:51.874301 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:52.468751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:52.468711 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:23:53.634020 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:53.633982 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:55.272468 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:23:55.272412 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:23:55.523616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:55.523531 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:56.290691 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:56.290299 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ssskh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:23:56.290691 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:56.290388 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" podUID="7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:23:57.214113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:57.213227 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:23:57.215114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:57.214504 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 02:23:57.253115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:57.237177 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:23:57.449364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:57.449318 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 02:23:57.471191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:57.459871 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:23:57.621896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:57.621850 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Oct 30 02:23:57.670343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:57.670300 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded" Oct 30 02:23:59.073035 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:23:59.072976 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:00.608027 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:00.607995 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:02.156497 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:02.156461 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:03.655564 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:03.655526 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:05.307839 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:05.307797 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:06.984998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:06.984953 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:08.587485 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:08.587435 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:10.191872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:10.191838 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:11.792915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:11.792883 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:13.039421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:13.039384 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4"] Oct 30 02:24:13.048169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:13.048139 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-400b9185-dhzc4"] Oct 30 02:24:13.358474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:13.358397 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:14.539761 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:14.539724 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="446df54f-cdf8-4604-a08b-6b9b603f370f" path="/var/lib/kubelet/pods/446df54f-cdf8-4604-a08b-6b9b603f370f/volumes" Oct 30 02:24:14.937998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:14.937926 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:16.488130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:16.488058 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:18.087450 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:18.087414 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:19.618077 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:19.618037 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:21.136831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:21.136783 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:21.264975 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:21.264761 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:21.392186 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:21.392069 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:22.799216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:22.799167 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:24.427260 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:24.427220 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:25.223315 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:24:25.223276 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:24:25.978016 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:25.977983 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:27.737166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:27.737130 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:28.955410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:28.955370 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc"] Oct 30 02:24:28.956028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:28.956012 5998 cpu_manager.go:414] "containerMap: removing container" podUID="032e9f87-821b-4408-9407-0ed3ce36dbfb" containerName="revoke-keys" Oct 30 02:24:28.956074 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:28.956029 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="032e9f87-821b-4408-9407-0ed3ce36dbfb" containerName="revoke-keys" Oct 30 02:24:28.956148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:28.956139 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="032e9f87-821b-4408-9407-0ed3ce36dbfb" containerName="revoke-keys" Oct 30 02:24:28.959848 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:28.959830 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc" Oct 30 02:24:28.983325 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:28.983295 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:24:28.983558 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:28.983309 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:24:28.983558 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:28.983309 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:24:28.983684 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:28.983375 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:24:28.991183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:28.991149 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc"] Oct 30 02:24:29.039916 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:29.039873 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfhn8\" (UniqueName: \"kubernetes.io/projected/a3b76a87-372a-4147-9b84-f1ef6048f0b3-kube-api-access-hfhn8\") pod \"maas-api-revoke-sub-644a4b02-hl9cc\" (UID: \"a3b76a87-372a-4147-9b84-f1ef6048f0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc" Oct 30 02:24:29.040234 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:29.040210 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b76a87-372a-4147-9b84-f1ef6048f0b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-644a4b02-hl9cc\" (UID: \"a3b76a87-372a-4147-9b84-f1ef6048f0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc" Oct 30 02:24:29.040337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:29.040268 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b76a87-372a-4147-9b84-f1ef6048f0b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-644a4b02-hl9cc\" (UID: \"a3b76a87-372a-4147-9b84-f1ef6048f0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc" Oct 30 02:24:29.141831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:29.141789 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfhn8\" (UniqueName: \"kubernetes.io/projected/a3b76a87-372a-4147-9b84-f1ef6048f0b3-kube-api-access-hfhn8\") pod \"maas-api-revoke-sub-644a4b02-hl9cc\" (UID: \"a3b76a87-372a-4147-9b84-f1ef6048f0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc" Oct 30 02:24:29.142051 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:29.142031 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b76a87-372a-4147-9b84-f1ef6048f0b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-644a4b02-hl9cc\" (UID: \"a3b76a87-372a-4147-9b84-f1ef6048f0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc" Oct 30 02:24:29.142153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:29.142136 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b76a87-372a-4147-9b84-f1ef6048f0b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-644a4b02-hl9cc\" (UID: \"a3b76a87-372a-4147-9b84-f1ef6048f0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc" Oct 30 02:24:29.142877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:29.142855 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b76a87-372a-4147-9b84-f1ef6048f0b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-644a4b02-hl9cc\" (UID: \"a3b76a87-372a-4147-9b84-f1ef6048f0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc" Oct 30 02:24:29.144534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:29.144510 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b76a87-372a-4147-9b84-f1ef6048f0b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-644a4b02-hl9cc\" (UID: \"a3b76a87-372a-4147-9b84-f1ef6048f0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc" Oct 30 02:24:29.164200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:29.164169 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfhn8\" (UniqueName: \"kubernetes.io/projected/a3b76a87-372a-4147-9b84-f1ef6048f0b3-kube-api-access-hfhn8\") pod \"maas-api-revoke-sub-644a4b02-hl9cc\" (UID: \"a3b76a87-372a-4147-9b84-f1ef6048f0b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc" Oct 30 02:24:29.277007 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:29.276911 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc" Oct 30 02:24:29.444079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:29.444035 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:29.613008 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:24:29.612970 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b76a87_372a_4147_9b84_f1ef6048f0b3.slice/crio-63050d700838bb57be639d8ba9a744a5ee8c872411ff0dd36dd124092ed6872e WatchSource:0}: Error finding container 63050d700838bb57be639d8ba9a744a5ee8c872411ff0dd36dd124092ed6872e: Status 404 returned error can't find the container with id 63050d700838bb57be639d8ba9a744a5ee8c872411ff0dd36dd124092ed6872e Oct 30 02:24:29.614156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:29.614125 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc"] Oct 30 02:24:29.885058 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:29.884972 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc" event={"ID":"a3b76a87-372a-4147-9b84-f1ef6048f0b3","Type":"ContainerStarted","Data":"63050d700838bb57be639d8ba9a744a5ee8c872411ff0dd36dd124092ed6872e"} Oct 30 02:24:30.894538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:30.894442 5998 generic.go:356] "Generic (PLEG): container finished" podID="a3b76a87-372a-4147-9b84-f1ef6048f0b3" containerID="969ed54ed780136ee97e43950a6c0a77f61572f0e4b9522b67f3020a6e2f126e" exitCode=0 Oct 30 02:24:30.894538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:30.894526 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc" event={"ID":"a3b76a87-372a-4147-9b84-f1ef6048f0b3","Type":"ContainerDied","Data":"969ed54ed780136ee97e43950a6c0a77f61572f0e4b9522b67f3020a6e2f126e"} Oct 30 02:24:31.069399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:31.069362 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:32.093074 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:32.093047 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc" Oct 30 02:24:32.197657 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:32.197619 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3b76a87-372a-4147-9b84-f1ef6048f0b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b76a87-372a-4147-9b84-f1ef6048f0b3-maas-api-openshift-ca\") pod \"a3b76a87-372a-4147-9b84-f1ef6048f0b3\" (UID: \"a3b76a87-372a-4147-9b84-f1ef6048f0b3\") " Oct 30 02:24:32.197872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:32.197723 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b76a87-372a-4147-9b84-f1ef6048f0b3-kube-api-access-hfhn8\" (UniqueName: \"kubernetes.io/projected/a3b76a87-372a-4147-9b84-f1ef6048f0b3-kube-api-access-hfhn8\") pod \"a3b76a87-372a-4147-9b84-f1ef6048f0b3\" (UID: \"a3b76a87-372a-4147-9b84-f1ef6048f0b3\") " Oct 30 02:24:32.197872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:32.197845 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3b76a87-372a-4147-9b84-f1ef6048f0b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b76a87-372a-4147-9b84-f1ef6048f0b3-maas-api-xks-ca\") pod \"a3b76a87-372a-4147-9b84-f1ef6048f0b3\" (UID: \"a3b76a87-372a-4147-9b84-f1ef6048f0b3\") " Oct 30 02:24:32.198034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:32.198006 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b76a87-372a-4147-9b84-f1ef6048f0b3-maas-api-openshift-ca" pod "a3b76a87-372a-4147-9b84-f1ef6048f0b3" (UID: "a3b76a87-372a-4147-9b84-f1ef6048f0b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:24:32.198428 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:32.198399 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b76a87-372a-4147-9b84-f1ef6048f0b3-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:24:32.199790 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:32.199769 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3b76a87-372a-4147-9b84-f1ef6048f0b3-maas-api-xks-ca" pod "a3b76a87-372a-4147-9b84-f1ef6048f0b3" (UID: "a3b76a87-372a-4147-9b84-f1ef6048f0b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:24:32.200033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:32.200001 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b76a87-372a-4147-9b84-f1ef6048f0b3-kube-api-access-hfhn8" pod "a3b76a87-372a-4147-9b84-f1ef6048f0b3" (UID: "a3b76a87-372a-4147-9b84-f1ef6048f0b3"). InnerVolumeSpecName "kube-api-access-hfhn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:24:32.300244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:32.300202 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b76a87-372a-4147-9b84-f1ef6048f0b3-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:24:32.300244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:32.300243 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfhn8\" (UniqueName: \"kubernetes.io/projected/a3b76a87-372a-4147-9b84-f1ef6048f0b3-kube-api-access-hfhn8\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:24:32.671877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:32.671840 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:32.912932 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:32.912901 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc" Oct 30 02:24:32.913121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:32.912932 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc" event={"ID":"a3b76a87-372a-4147-9b84-f1ef6048f0b3","Type":"ContainerDied","Data":"63050d700838bb57be639d8ba9a744a5ee8c872411ff0dd36dd124092ed6872e"} Oct 30 02:24:32.913121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:32.912992 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63050d700838bb57be639d8ba9a744a5ee8c872411ff0dd36dd124092ed6872e" Oct 30 02:24:33.464301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:33.464262 5998 scope.go:122] "RemoveContainer" containerID="2742f2a02b2d5b241a089af1d2dd9a52598c53dafa4017c13086f7075bdd2bd4" Oct 30 02:24:34.289744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:34.289710 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:35.033656 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:35.033614 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm"] Oct 30 02:24:35.043495 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:35.043453 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afdfb4f7-nbklm"] Oct 30 02:24:35.780286 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:35.780251 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:36.541814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:36.541771 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c0c20b-4385-42e0-83e9-bdb700eee0b3" path="/var/lib/kubelet/pods/59c0c20b-4385-42e0-83e9-bdb700eee0b3/volumes" Oct 30 02:24:37.398896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:37.398861 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:38.746757 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:38.746714 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:39.003260 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:39.003175 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:39.801372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:39.801337 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c"] Oct 30 02:24:39.801964 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:39.801950 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a3b76a87-372a-4147-9b84-f1ef6048f0b3" containerName="revoke-keys" Oct 30 02:24:39.801964 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:39.801965 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b76a87-372a-4147-9b84-f1ef6048f0b3" containerName="revoke-keys" Oct 30 02:24:39.802082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:39.802069 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b76a87-372a-4147-9b84-f1ef6048f0b3" containerName="revoke-keys" Oct 30 02:24:39.807028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:39.807008 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c" Oct 30 02:24:39.822592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:39.822569 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:24:39.832807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:39.832752 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:24:39.832807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:39.832797 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:24:39.833030 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:39.832950 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:24:39.839747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:39.839715 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c"] Oct 30 02:24:39.915434 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:39.915394 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-799fdeb2-mhj6c\" (UID: \"dcbc3c28-015c-4c5d-8c02-02ffb6becda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c" Oct 30 02:24:39.915632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:39.915581 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvh69\" (UniqueName: \"kubernetes.io/projected/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-kube-api-access-nvh69\") pod \"maas-api-revoke-sub-799fdeb2-mhj6c\" (UID: \"dcbc3c28-015c-4c5d-8c02-02ffb6becda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c" Oct 30 02:24:39.915718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:39.915703 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-799fdeb2-mhj6c\" (UID: \"dcbc3c28-015c-4c5d-8c02-02ffb6becda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c" Oct 30 02:24:40.016935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:40.016893 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-799fdeb2-mhj6c\" (UID: \"dcbc3c28-015c-4c5d-8c02-02ffb6becda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c" Oct 30 02:24:40.017166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:40.017011 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvh69\" (UniqueName: \"kubernetes.io/projected/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-kube-api-access-nvh69\") pod \"maas-api-revoke-sub-799fdeb2-mhj6c\" (UID: \"dcbc3c28-015c-4c5d-8c02-02ffb6becda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c" Oct 30 02:24:40.017166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:40.017132 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-799fdeb2-mhj6c\" (UID: \"dcbc3c28-015c-4c5d-8c02-02ffb6becda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c" Oct 30 02:24:40.017787 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:40.017756 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-799fdeb2-mhj6c\" (UID: \"dcbc3c28-015c-4c5d-8c02-02ffb6becda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c" Oct 30 02:24:40.019618 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:40.019590 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-799fdeb2-mhj6c\" (UID: \"dcbc3c28-015c-4c5d-8c02-02ffb6becda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c" Oct 30 02:24:40.039771 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:40.039742 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvh69\" (UniqueName: \"kubernetes.io/projected/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-kube-api-access-nvh69\") pod \"maas-api-revoke-sub-799fdeb2-mhj6c\" (UID: \"dcbc3c28-015c-4c5d-8c02-02ffb6becda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c" Oct 30 02:24:40.123907 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:40.123823 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c" Oct 30 02:24:40.433651 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:40.433622 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c"] Oct 30 02:24:40.437067 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:24:40.437034 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbc3c28_015c_4c5d_8c02_02ffb6becda3.slice/crio-c562b8e18f07aca85877aea68ca1a86937a27ef0b94bd63c435ab497f83278b8 WatchSource:0}: Error finding container c562b8e18f07aca85877aea68ca1a86937a27ef0b94bd63c435ab497f83278b8: Status 404 returned error can't find the container with id c562b8e18f07aca85877aea68ca1a86937a27ef0b94bd63c435ab497f83278b8 Oct 30 02:24:40.795926 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:40.795843 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:40.981839 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:40.981799 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c" event={"ID":"dcbc3c28-015c-4c5d-8c02-02ffb6becda3","Type":"ContainerStarted","Data":"8c2423a97b13e463848db9518cc2dc9fb4c157c3406355025b69677720442d71"} Oct 30 02:24:40.982347 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:40.981856 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c" event={"ID":"dcbc3c28-015c-4c5d-8c02-02ffb6becda3","Type":"ContainerStarted","Data":"c562b8e18f07aca85877aea68ca1a86937a27ef0b94bd63c435ab497f83278b8"} Oct 30 02:24:41.020215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:41.020156 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c" podStartSLOduration=2.020135097 podStartE2EDuration="2.020135097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:24:41.018888112 +0000 UTC m=+3310.597679193" watchObservedRunningTime="2026-10-30 02:24:41.020135097 +0000 UTC m=+3310.598926179" Oct 30 02:24:41.992451 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:41.992409 5998 generic.go:356] "Generic (PLEG): container finished" podID="dcbc3c28-015c-4c5d-8c02-02ffb6becda3" containerID="8c2423a97b13e463848db9518cc2dc9fb4c157c3406355025b69677720442d71" exitCode=0 Oct 30 02:24:41.992930 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:41.992488 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c" event={"ID":"dcbc3c28-015c-4c5d-8c02-02ffb6becda3","Type":"ContainerDied","Data":"8c2423a97b13e463848db9518cc2dc9fb4c157c3406355025b69677720442d71"} Oct 30 02:24:42.491767 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:42.491729 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:43.171985 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:43.171958 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c" Oct 30 02:24:43.278291 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:43.278203 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-maas-api-xks-ca\") pod \"dcbc3c28-015c-4c5d-8c02-02ffb6becda3\" (UID: \"dcbc3c28-015c-4c5d-8c02-02ffb6becda3\") " Oct 30 02:24:43.278463 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:43.278332 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-maas-api-openshift-ca\") pod \"dcbc3c28-015c-4c5d-8c02-02ffb6becda3\" (UID: \"dcbc3c28-015c-4c5d-8c02-02ffb6becda3\") " Oct 30 02:24:43.278688 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:43.278665 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-kube-api-access-nvh69\" (UniqueName: \"kubernetes.io/projected/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-kube-api-access-nvh69\") pod \"dcbc3c28-015c-4c5d-8c02-02ffb6becda3\" (UID: \"dcbc3c28-015c-4c5d-8c02-02ffb6becda3\") " Oct 30 02:24:43.278793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:43.278683 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-maas-api-openshift-ca" pod "dcbc3c28-015c-4c5d-8c02-02ffb6becda3" (UID: "dcbc3c28-015c-4c5d-8c02-02ffb6becda3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:24:43.279184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:43.279162 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:24:43.280431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:43.280408 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-maas-api-xks-ca" pod "dcbc3c28-015c-4c5d-8c02-02ffb6becda3" (UID: "dcbc3c28-015c-4c5d-8c02-02ffb6becda3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:24:43.280801 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:43.280778 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-kube-api-access-nvh69" pod "dcbc3c28-015c-4c5d-8c02-02ffb6becda3" (UID: "dcbc3c28-015c-4c5d-8c02-02ffb6becda3"). InnerVolumeSpecName "kube-api-access-nvh69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:24:43.381023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:43.380988 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvh69\" (UniqueName: \"kubernetes.io/projected/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-kube-api-access-nvh69\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:24:43.381023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:43.381016 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcbc3c28-015c-4c5d-8c02-02ffb6becda3-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:24:44.009338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:44.009299 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c" event={"ID":"dcbc3c28-015c-4c5d-8c02-02ffb6becda3","Type":"ContainerDied","Data":"c562b8e18f07aca85877aea68ca1a86937a27ef0b94bd63c435ab497f83278b8"} Oct 30 02:24:44.009338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:44.009339 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c" Oct 30 02:24:44.009559 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:44.009354 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c562b8e18f07aca85877aea68ca1a86937a27ef0b94bd63c435ab497f83278b8" Oct 30 02:24:44.083943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:44.083890 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:44.603115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:44.602669 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:24:45.920113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:45.915690 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:47.037523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:47.037487 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh"] Oct 30 02:24:47.048526 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:47.048481 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx"] Oct 30 02:24:47.053628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:47.053593 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9d99c15-lh5mx"] Oct 30 02:24:47.059033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:47.059006 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a096a0d9-dwpqh"] Oct 30 02:24:47.679128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:47.679071 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:48.542791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:48.542753 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b1225a9-cd64-4a55-9051-a7fa3a16b3e7" path="/var/lib/kubelet/pods/1b1225a9-cd64-4a55-9051-a7fa3a16b3e7/volumes" Oct 30 02:24:48.543367 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:48.543349 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230d086e-bfe5-4c4a-805b-404cd9c5cba7" path="/var/lib/kubelet/pods/230d086e-bfe5-4c4a-805b-404cd9c5cba7/volumes" Oct 30 02:24:49.410861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:49.410803 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:51.061338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:51.061269 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:51.180106 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:51.179910 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:52.165272 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:52.165231 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4"] Oct 30 02:24:52.167427 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:52.167390 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:52.518318 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:52.518193 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:52.870183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:52.869134 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:53.302188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:53.302144 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:54.876247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:54.876212 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:55.220253 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:24:55.220218 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:24:56.566050 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:56.566007 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:57.165738 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:57.165687 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" podUID="8a70ffed-6e43-4b01-9bb8-7973c3c907e1" containerName="payload-processing" containerID="cri-o://539d12177b4a6c0c9ae8c0f3d9dd9621ec482dca3d7a36c51214df217dcce047" gracePeriod=25 Oct 30 02:24:57.439104 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:57.439063 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" Oct 30 02:24:57.443296 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:57.443271 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-kube-api-access-596df\" (UniqueName: \"kubernetes.io/projected/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-kube-api-access-596df\") pod \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\" (UID: \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\") " Oct 30 02:24:57.443434 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:57.443404 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-provider-credentials\") pod \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\" (UID: \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\") " Oct 30 02:24:57.443554 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:57.443537 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-routing-overlay-volume\") pod \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\" (UID: \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\") " Oct 30 02:24:57.443652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:57.443636 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-plugins-config-volume\") pod \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\" (UID: \"8a70ffed-6e43-4b01-9bb8-7973c3c907e1\") " Oct 30 02:24:57.443854 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:57.443828 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-routing-overlay-volume" pod "8a70ffed-6e43-4b01-9bb8-7973c3c907e1" (UID: "8a70ffed-6e43-4b01-9bb8-7973c3c907e1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:24:57.444107 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:57.444070 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-plugins-config-volume" pod "8a70ffed-6e43-4b01-9bb8-7973c3c907e1" (UID: "8a70ffed-6e43-4b01-9bb8-7973c3c907e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:24:57.447162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:57.444284 5998 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-routing-overlay-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:24:57.447162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:57.445633 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-provider-credentials" pod "8a70ffed-6e43-4b01-9bb8-7973c3c907e1" (UID: "8a70ffed-6e43-4b01-9bb8-7973c3c907e1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:24:57.447162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:57.445711 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-kube-api-access-596df" pod "8a70ffed-6e43-4b01-9bb8-7973c3c907e1" (UID: "8a70ffed-6e43-4b01-9bb8-7973c3c907e1"). InnerVolumeSpecName "kube-api-access-596df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:24:57.549357 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:57.549316 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:24:57.549357 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:57.549356 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-596df\" (UniqueName: \"kubernetes.io/projected/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-kube-api-access-596df\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:24:57.549628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:57.549373 5998 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a70ffed-6e43-4b01-9bb8-7973c3c907e1-provider-credentials\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:24:58.137190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:58.137152 5998 generic.go:356] "Generic (PLEG): container finished" podID="8a70ffed-6e43-4b01-9bb8-7973c3c907e1" containerID="539d12177b4a6c0c9ae8c0f3d9dd9621ec482dca3d7a36c51214df217dcce047" exitCode=0 Oct 30 02:24:58.137664 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:58.137215 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" event={"ID":"8a70ffed-6e43-4b01-9bb8-7973c3c907e1","Type":"ContainerDied","Data":"539d12177b4a6c0c9ae8c0f3d9dd9621ec482dca3d7a36c51214df217dcce047"} Oct 30 02:24:58.137664 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:58.137238 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" Oct 30 02:24:58.137664 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:58.137277 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4" event={"ID":"8a70ffed-6e43-4b01-9bb8-7973c3c907e1","Type":"ContainerDied","Data":"c284bbc29a522b26093505464025e591c26a5d5d25ec16fd1c8ce933f110cba8"} Oct 30 02:24:58.137664 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:58.137304 5998 scope.go:122] "RemoveContainer" containerID="539d12177b4a6c0c9ae8c0f3d9dd9621ec482dca3d7a36c51214df217dcce047" Oct 30 02:24:58.149495 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:58.149437 5998 scope.go:122] "RemoveContainer" containerID="539d12177b4a6c0c9ae8c0f3d9dd9621ec482dca3d7a36c51214df217dcce047" Oct 30 02:24:58.149872 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:24:58.149835 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"539d12177b4a6c0c9ae8c0f3d9dd9621ec482dca3d7a36c51214df217dcce047\": container with ID starting with 539d12177b4a6c0c9ae8c0f3d9dd9621ec482dca3d7a36c51214df217dcce047 not found: ID does not exist" containerID="539d12177b4a6c0c9ae8c0f3d9dd9621ec482dca3d7a36c51214df217dcce047" Oct 30 02:24:58.150620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:58.149877 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"539d12177b4a6c0c9ae8c0f3d9dd9621ec482dca3d7a36c51214df217dcce047"} err="failed to get container status \"539d12177b4a6c0c9ae8c0f3d9dd9621ec482dca3d7a36c51214df217dcce047\": rpc error: code = NotFound desc = could not find container \"539d12177b4a6c0c9ae8c0f3d9dd9621ec482dca3d7a36c51214df217dcce047\": container with ID starting with 539d12177b4a6c0c9ae8c0f3d9dd9621ec482dca3d7a36c51214df217dcce047 not found: ID does not exist" Oct 30 02:24:58.186651 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:58.186618 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4"] Oct 30 02:24:58.196199 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:58.196173 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-95f40f/payload-processing-external-model-e2e-extmod-a-95f40f-54d6jrls4"] Oct 30 02:24:58.310576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:58.310536 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:24:58.539745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:58.539707 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a70ffed-6e43-4b01-9bb8-7973c3c907e1" path="/var/lib/kubelet/pods/8a70ffed-6e43-4b01-9bb8-7973c3c907e1/volumes" Oct 30 02:24:59.043918 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:59.043876 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp"] Oct 30 02:24:59.049708 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:24:59.049676 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-006df12a-ghvtp"] Oct 30 02:25:00.072945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:00.072911 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:00.324485 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:00.324395 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:00.439744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:00.439712 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:00.540998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:00.540951 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ec49c9-f31e-4e0f-ab12-599eb9121f8c" path="/var/lib/kubelet/pods/08ec49c9-f31e-4e0f-ab12-599eb9121f8c/volumes" Oct 30 02:25:00.551432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:00.551402 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:01.808790 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:01.808754 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:02.397198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:02.397168 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:03.471915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:03.471869 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:05.004859 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:05.004812 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:06.581064 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:06.581010 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:07.891865 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:07.891817 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:08.085595 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:08.085538 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:08.367820 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:08.367779 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:08.493963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:08.493927 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:09.180114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.173343 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999"] Oct 30 02:25:09.198746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.198701 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8a70ffed-6e43-4b01-9bb8-7973c3c907e1" containerName="payload-processing" Oct 30 02:25:09.199015 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.199000 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a70ffed-6e43-4b01-9bb8-7973c3c907e1" containerName="payload-processing" Oct 30 02:25:09.199164 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.199153 5998 cpu_manager.go:414] "containerMap: removing container" podUID="dcbc3c28-015c-4c5d-8c02-02ffb6becda3" containerName="revoke-keys" Oct 30 02:25:09.199263 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.199253 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbc3c28-015c-4c5d-8c02-02ffb6becda3" containerName="revoke-keys" Oct 30 02:25:09.201508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.201482 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a70ffed-6e43-4b01-9bb8-7973c3c907e1" containerName="payload-processing" Oct 30 02:25:09.205112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.201688 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbc3c28-015c-4c5d-8c02-02ffb6becda3" containerName="revoke-keys" Oct 30 02:25:09.328560 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.328526 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.328811 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.328789 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf65846d-6629-424c-9eda-4c359fd1992a-istiod-ca-cert\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.328982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.328956 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf65846d-6629-424c-9eda-4c359fd1992a-istio-token\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.329143 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.329125 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7kv4\" (UniqueName: \"kubernetes.io/projected/bf65846d-6629-424c-9eda-4c359fd1992a-kube-api-access-p7kv4\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.329322 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.329297 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-workload-socket\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.329473 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.329456 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-istio-envoy\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.329730 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.329702 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-istio-data\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.329904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.329883 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf65846d-6629-424c-9eda-4c359fd1992a-istio-podinfo\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.330145 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.330127 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-workload-certs\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.330302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.330287 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-credential-socket\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.411235 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.409948 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999"] Oct 30 02:25:09.487827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.487663 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2943ff-openshift-default-dockercfg-92vm7\"" Oct 30 02:25:09.494753 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.493815 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-istio-envoy\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.494753 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.493896 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-istio-data\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.494753 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.493936 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf65846d-6629-424c-9eda-4c359fd1992a-istio-podinfo\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.494753 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.494028 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-workload-certs\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.494753 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.494061 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-credential-socket\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.495252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.494809 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-istio-data\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.498133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.495954 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf65846d-6629-424c-9eda-4c359fd1992a-istiod-ca-cert\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.498133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.496017 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf65846d-6629-424c-9eda-4c359fd1992a-istio-token\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.498133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.496065 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7kv4\" (UniqueName: \"kubernetes.io/projected/bf65846d-6629-424c-9eda-4c359fd1992a-kube-api-access-p7kv4\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.498133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.496140 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-workload-socket\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.498133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.496506 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-workload-socket\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.500582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.499394 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-credential-socket\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.500582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.499643 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-workload-certs\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.500582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.500509 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf65846d-6629-424c-9eda-4c359fd1992a-istiod-ca-cert\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.509221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.508324 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:09.536996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.536903 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf65846d-6629-424c-9eda-4c359fd1992a-istio-podinfo\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.573063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.564337 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7kv4\" (UniqueName: \"kubernetes.io/projected/bf65846d-6629-424c-9eda-4c359fd1992a-kube-api-access-p7kv4\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.574672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.574573 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-istio-envoy\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.575140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.575118 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf65846d-6629-424c-9eda-4c359fd1992a-istio-token\") pod \"e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:09.628532 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.628495 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:09.871161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:09.870489 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:10.301812 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:25:10.301769 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf65846d_6629_424c_9eda_4c359fd1992a.slice/crio-99295b5c40a88c10aebfec2825aed110deedcacdc2748b0f8dc6a062b234bc32 WatchSource:0}: Error finding container 99295b5c40a88c10aebfec2825aed110deedcacdc2748b0f8dc6a062b234bc32: Status 404 returned error can't find the container with id 99295b5c40a88c10aebfec2825aed110deedcacdc2748b0f8dc6a062b234bc32 Oct 30 02:25:10.307574 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:10.306304 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:25:10.307574 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:10.306404 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:25:10.307574 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:10.306450 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:25:10.319343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:10.319317 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999"] Oct 30 02:25:10.582658 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:10.582134 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" event={"ID":"bf65846d-6629-424c-9eda-4c359fd1992a","Type":"ContainerStarted","Data":"99295b5c40a88c10aebfec2825aed110deedcacdc2748b0f8dc6a062b234bc32"} Oct 30 02:25:10.769115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:10.752427 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:10.908184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:10.905613 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:11.313120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:11.311876 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:11.662822 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:11.661701 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" event={"ID":"bf65846d-6629-424c-9eda-4c359fd1992a","Type":"ContainerStarted","Data":"8ee63260b4416374d92f1a6eb0f58f9d8fb329fb304a04685a8d2622b454fca2"} Oct 30 02:25:11.766472 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:11.766200 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" podStartSLOduration=2.766177161 podStartE2EDuration="2.766177161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:25:11.745670571 +0000 UTC m=+3341.324461652" watchObservedRunningTime="2026-10-30 02:25:11.766177161 +0000 UTC m=+3341.344968245" Oct 30 02:25:11.877297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:11.876697 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:11.897122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:11.895569 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:11.936612 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:11.936515 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj"] Oct 30 02:25:12.021113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:11.983303 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.021113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.014532 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d97e4c-openshift-default-dockercfg-hjfgz\"" Oct 30 02:25:12.070987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.070945 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:12.196085 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.195971 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-credential-socket\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.206294 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.206249 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-data\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.207152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.207120 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxdkm\" (UniqueName: \"kubernetes.io/projected/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-kube-api-access-rxdkm\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.207849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.207824 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istiod-ca-cert\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.220274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.220222 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-workload-certs\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.220609 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.220586 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-token\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.220927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.220889 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-envoy\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.221179 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.221150 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-podinfo\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.221460 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.221420 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-workload-socket\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.314567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.314478 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:12.354156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.331262 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-token\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.354156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.331342 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-envoy\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.354156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.331394 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-podinfo\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.354156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.331437 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-workload-socket\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.354156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.331524 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-credential-socket\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.354156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.331568 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-data\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.354156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.331592 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxdkm\" (UniqueName: \"kubernetes.io/projected/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-kube-api-access-rxdkm\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.354156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.331797 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istiod-ca-cert\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.354156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.331828 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-workload-certs\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.354156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.332313 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-workload-certs\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.398118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.374558 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-credential-socket\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.406893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.398936 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-data\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.464247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.426856 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-workload-socket\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.464247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.427193 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-envoy\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.464247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.428428 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istiod-ca-cert\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.464247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.443220 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj"] Oct 30 02:25:12.494337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.494270 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-podinfo\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:12.689850 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:12.689813 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-token\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:13.002849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.002623 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxdkm\" (UniqueName: \"kubernetes.io/projected/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-kube-api-access-rxdkm\") pod \"e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:13.063111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.041522 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:13.151978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.151938 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj"] Oct 30 02:25:13.178525 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.178481 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:25:13.199124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.199076 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:13.257358 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.257281 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:13.257615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.257598 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj"] Oct 30 02:25:13.257793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.257780 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:25:13.258282 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.258260 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj" Oct 30 02:25:13.289046 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.289014 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:25:13.320140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.316574 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:25:13.320140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.316677 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:25:13.320140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.316790 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:25:13.320140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.319014 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qzsh\" (UniqueName: \"kubernetes.io/projected/fbb9e882-4da2-4472-95ae-84ac0d5d419e-kube-api-access-2qzsh\") pod \"maas-api-revoke-sub-11f0a78f-wvzjj\" (UID: \"fbb9e882-4da2-4472-95ae-84ac0d5d419e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj" Oct 30 02:25:13.320140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.319218 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb9e882-4da2-4472-95ae-84ac0d5d419e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11f0a78f-wvzjj\" (UID: \"fbb9e882-4da2-4472-95ae-84ac0d5d419e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj" Oct 30 02:25:13.320140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.319302 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb9e882-4da2-4472-95ae-84ac0d5d419e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11f0a78f-wvzjj\" (UID: \"fbb9e882-4da2-4472-95ae-84ac0d5d419e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj" Oct 30 02:25:13.344209 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.344167 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-jhqft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:25:13.344525 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.344489 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" podUID="f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:25:13.578319 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.552812 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb9e882-4da2-4472-95ae-84ac0d5d419e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11f0a78f-wvzjj\" (UID: \"fbb9e882-4da2-4472-95ae-84ac0d5d419e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj" Oct 30 02:25:13.578319 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.552954 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb9e882-4da2-4472-95ae-84ac0d5d419e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11f0a78f-wvzjj\" (UID: \"fbb9e882-4da2-4472-95ae-84ac0d5d419e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj" Oct 30 02:25:13.578319 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.554287 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qzsh\" (UniqueName: \"kubernetes.io/projected/fbb9e882-4da2-4472-95ae-84ac0d5d419e-kube-api-access-2qzsh\") pod \"maas-api-revoke-sub-11f0a78f-wvzjj\" (UID: \"fbb9e882-4da2-4472-95ae-84ac0d5d419e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj" Oct 30 02:25:13.578319 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.555787 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb9e882-4da2-4472-95ae-84ac0d5d419e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11f0a78f-wvzjj\" (UID: \"fbb9e882-4da2-4472-95ae-84ac0d5d419e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj" Oct 30 02:25:13.578319 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.560715 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb9e882-4da2-4472-95ae-84ac0d5d419e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11f0a78f-wvzjj\" (UID: \"fbb9e882-4da2-4472-95ae-84ac0d5d419e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj" Oct 30 02:25:13.578319 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.560783 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded" Oct 30 02:25:13.614167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.598692 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qzsh\" (UniqueName: \"kubernetes.io/projected/fbb9e882-4da2-4472-95ae-84ac0d5d419e-kube-api-access-2qzsh\") pod \"maas-api-revoke-sub-11f0a78f-wvzjj\" (UID: \"fbb9e882-4da2-4472-95ae-84ac0d5d419e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj" Oct 30 02:25:13.614167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.600520 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452"] Oct 30 02:25:13.614412 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.614231 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452"] Oct 30 02:25:13.614412 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.614342 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452" Oct 30 02:25:13.702114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.689848 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj" Oct 30 02:25:13.761177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.760783 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f255f8f-tv452\" (UID: \"d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452" Oct 30 02:25:13.761177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.760892 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f255f8f-tv452\" (UID: \"d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452" Oct 30 02:25:13.761177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.761053 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhn7d\" (UniqueName: \"kubernetes.io/projected/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-kube-api-access-lhn7d\") pod \"maas-api-revoke-sub-9f255f8f-tv452\" (UID: \"d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452" Oct 30 02:25:13.934619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.931815 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhn7d\" (UniqueName: \"kubernetes.io/projected/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-kube-api-access-lhn7d\") pod \"maas-api-revoke-sub-9f255f8f-tv452\" (UID: \"d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452" Oct 30 02:25:13.934619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.932047 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f255f8f-tv452\" (UID: \"d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452" Oct 30 02:25:13.934619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.932155 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f255f8f-tv452\" (UID: \"d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452" Oct 30 02:25:14.005115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:13.967869 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f255f8f-tv452\" (UID: \"d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452" Oct 30 02:25:14.058278 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:14.043656 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhn7d\" (UniqueName: \"kubernetes.io/projected/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-kube-api-access-lhn7d\") pod \"maas-api-revoke-sub-9f255f8f-tv452\" (UID: \"d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452" Oct 30 02:25:14.100485 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:14.080430 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f255f8f-tv452\" (UID: \"d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452" Oct 30 02:25:14.278709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:14.218997 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj"] Oct 30 02:25:14.322010 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:25:14.321958 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1e7fb2_51e7_442a_ae7f_e19d9671c65b.slice/crio-9c18230d5a068797c98d44cfe7c74e6b1c126fee4ace7d387ee715a283d7df4b WatchSource:0}: Error finding container 9c18230d5a068797c98d44cfe7c74e6b1c126fee4ace7d387ee715a283d7df4b: Status 404 returned error can't find the container with id 9c18230d5a068797c98d44cfe7c74e6b1c126fee4ace7d387ee715a283d7df4b Oct 30 02:25:14.361398 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:14.352778 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:25:14.361398 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:14.352879 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:25:14.361398 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:14.352918 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:25:14.361398 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:14.353553 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452" Oct 30 02:25:14.468193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:14.467987 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:14.555591 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:14.553576 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj"] Oct 30 02:25:14.567543 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:25:14.567481 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb9e882_4da2_4472_95ae_84ac0d5d419e.slice/crio-efbaae92b11293c3f9758da7d77ef33cb995c4c118be950769e74f929e110bd1 WatchSource:0}: Error finding container efbaae92b11293c3f9758da7d77ef33cb995c4c118be950769e74f929e110bd1: Status 404 returned error can't find the container with id efbaae92b11293c3f9758da7d77ef33cb995c4c118be950769e74f929e110bd1 Oct 30 02:25:14.651154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:14.649565 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:14.813208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:14.813181 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452"] Oct 30 02:25:14.816548 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:25:14.816501 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ae4d4a_eca4_4ddb_a316_9d9ab9598bed.slice/crio-bbe51da83fc2cb3b8b69973d4f8894e0def0df0a02f489391214ca3b08a14778 WatchSource:0}: Error finding container bbe51da83fc2cb3b8b69973d4f8894e0def0df0a02f489391214ca3b08a14778: Status 404 returned error can't find the container with id bbe51da83fc2cb3b8b69973d4f8894e0def0df0a02f489391214ca3b08a14778 Oct 30 02:25:14.879551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:14.879514 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:15.219042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:15.219005 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" event={"ID":"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b","Type":"ContainerStarted","Data":"2d7ea69ae41a5967fd68e4dc2bc2f12c95f6a766bd7de576d29f7b03492bbe5e"} Oct 30 02:25:15.219819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:15.219078 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" event={"ID":"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b","Type":"ContainerStarted","Data":"9c18230d5a068797c98d44cfe7c74e6b1c126fee4ace7d387ee715a283d7df4b"} Oct 30 02:25:15.222537 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:15.222497 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452" event={"ID":"d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed","Type":"ContainerStarted","Data":"bbe51da83fc2cb3b8b69973d4f8894e0def0df0a02f489391214ca3b08a14778"} Oct 30 02:25:15.224436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:15.224389 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj" event={"ID":"fbb9e882-4da2-4472-95ae-84ac0d5d419e","Type":"ContainerStarted","Data":"70df15320129a966040eb602df406a26903b0d4ccdb8f4c817ecd8923cf2a129"} Oct 30 02:25:15.224543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:15.224460 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj" event={"ID":"fbb9e882-4da2-4472-95ae-84ac0d5d419e","Type":"ContainerStarted","Data":"efbaae92b11293c3f9758da7d77ef33cb995c4c118be950769e74f929e110bd1"} Oct 30 02:25:15.270873 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:15.270815 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" podStartSLOduration=4.270800694 podStartE2EDuration="4.270800694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:25:15.265141768 +0000 UTC m=+3344.843932849" watchObservedRunningTime="2026-10-30 02:25:15.270800694 +0000 UTC m=+3344.849591775" Oct 30 02:25:15.308303 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:15.308251 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj" podStartSLOduration=3.3082379 podStartE2EDuration="3.3082379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:25:15.302569093 +0000 UTC m=+3344.881360175" watchObservedRunningTime="2026-10-30 02:25:15.3082379 +0000 UTC m=+3344.887028980" Oct 30 02:25:16.200774 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:16.200744 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:16.208678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:16.208646 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:16.233995 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:16.233962 5998 generic.go:356] "Generic (PLEG): container finished" podID="d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed" containerID="a69190d8cd0c0bc2b4da998d4f86bdf46360c5641ba82c1d77de5c2896f04d9e" exitCode=0 Oct 30 02:25:16.234188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:16.234045 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452" event={"ID":"d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed","Type":"ContainerDied","Data":"a69190d8cd0c0bc2b4da998d4f86bdf46360c5641ba82c1d77de5c2896f04d9e"} Oct 30 02:25:16.236036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:16.235995 5998 generic.go:356] "Generic (PLEG): container finished" podID="fbb9e882-4da2-4472-95ae-84ac0d5d419e" containerID="70df15320129a966040eb602df406a26903b0d4ccdb8f4c817ecd8923cf2a129" exitCode=0 Oct 30 02:25:16.236523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:16.236444 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj" event={"ID":"fbb9e882-4da2-4472-95ae-84ac0d5d419e","Type":"ContainerDied","Data":"70df15320129a966040eb602df406a26903b0d4ccdb8f4c817ecd8923cf2a129"} Oct 30 02:25:16.236832 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:16.236814 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:16.237810 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:16.237793 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:25:16.242371 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:16.242211 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:16.374159 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:16.374123 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:16.832451 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:16.828918 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:17.461383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.461350 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj" Oct 30 02:25:17.465215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.465188 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452" Oct 30 02:25:17.550850 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.550757 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-maas-api-xks-ca\") pod \"d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed\" (UID: \"d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed\") " Oct 30 02:25:17.550850 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.550839 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-kube-api-access-lhn7d\" (UniqueName: \"kubernetes.io/projected/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-kube-api-access-lhn7d\") pod \"d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed\" (UID: \"d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed\") " Oct 30 02:25:17.551125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.550989 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-maas-api-openshift-ca\") pod \"d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed\" (UID: \"d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed\") " Oct 30 02:25:17.551125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.551042 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb9e882-4da2-4472-95ae-84ac0d5d419e-kube-api-access-2qzsh\" (UniqueName: \"kubernetes.io/projected/fbb9e882-4da2-4472-95ae-84ac0d5d419e-kube-api-access-2qzsh\") pod \"fbb9e882-4da2-4472-95ae-84ac0d5d419e\" (UID: \"fbb9e882-4da2-4472-95ae-84ac0d5d419e\") " Oct 30 02:25:17.551271 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.551166 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbb9e882-4da2-4472-95ae-84ac0d5d419e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb9e882-4da2-4472-95ae-84ac0d5d419e-maas-api-xks-ca\") pod \"fbb9e882-4da2-4472-95ae-84ac0d5d419e\" (UID: \"fbb9e882-4da2-4472-95ae-84ac0d5d419e\") " Oct 30 02:25:17.551271 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.551232 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb9e882-4da2-4472-95ae-84ac0d5d419e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb9e882-4da2-4472-95ae-84ac0d5d419e-maas-api-openshift-ca\") pod \"fbb9e882-4da2-4472-95ae-84ac0d5d419e\" (UID: \"fbb9e882-4da2-4472-95ae-84ac0d5d419e\") " Oct 30 02:25:17.552374 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.552343 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-maas-api-openshift-ca" pod "d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed" (UID: "d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:25:17.553321 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.553265 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb9e882-4da2-4472-95ae-84ac0d5d419e-maas-api-openshift-ca" pod "fbb9e882-4da2-4472-95ae-84ac0d5d419e" (UID: "fbb9e882-4da2-4472-95ae-84ac0d5d419e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:25:17.557018 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.556960 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-maas-api-xks-ca" pod "d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed" (UID: "d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:25:17.557018 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.556984 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-kube-api-access-lhn7d" pod "d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed" (UID: "d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed"). InnerVolumeSpecName "kube-api-access-lhn7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:25:17.557314 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.557282 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb9e882-4da2-4472-95ae-84ac0d5d419e-kube-api-access-2qzsh" pod "fbb9e882-4da2-4472-95ae-84ac0d5d419e" (UID: "fbb9e882-4da2-4472-95ae-84ac0d5d419e"). InnerVolumeSpecName "kube-api-access-2qzsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:25:17.558491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.558467 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb9e882-4da2-4472-95ae-84ac0d5d419e-maas-api-xks-ca" pod "fbb9e882-4da2-4472-95ae-84ac0d5d419e" (UID: "fbb9e882-4da2-4472-95ae-84ac0d5d419e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:25:17.653709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.653671 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:25:17.653709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.653704 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhn7d\" (UniqueName: \"kubernetes.io/projected/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-kube-api-access-lhn7d\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:25:17.653709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.653715 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:25:17.654042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.653723 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qzsh\" (UniqueName: \"kubernetes.io/projected/fbb9e882-4da2-4472-95ae-84ac0d5d419e-kube-api-access-2qzsh\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:25:17.654042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.653734 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb9e882-4da2-4472-95ae-84ac0d5d419e-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:25:17.654042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.653742 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb9e882-4da2-4472-95ae-84ac0d5d419e-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:25:17.823140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.823052 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:17.987230 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:17.987189 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:18.106175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:18.104150 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx"] Oct 30 02:25:18.191431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:18.154306 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ba2fbb-s98dx"] Oct 30 02:25:18.316849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:18.316812 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452" event={"ID":"d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed","Type":"ContainerDied","Data":"bbe51da83fc2cb3b8b69973d4f8894e0def0df0a02f489391214ca3b08a14778"} Oct 30 02:25:18.317476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:18.317192 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452" Oct 30 02:25:18.317696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:18.317677 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbe51da83fc2cb3b8b69973d4f8894e0def0df0a02f489391214ca3b08a14778" Oct 30 02:25:18.321138 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:18.321121 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj" Oct 30 02:25:18.331460 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:18.331430 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj" event={"ID":"fbb9e882-4da2-4472-95ae-84ac0d5d419e","Type":"ContainerDied","Data":"efbaae92b11293c3f9758da7d77ef33cb995c4c118be950769e74f929e110bd1"} Oct 30 02:25:18.331704 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:18.331686 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efbaae92b11293c3f9758da7d77ef33cb995c4c118be950769e74f929e110bd1" Oct 30 02:25:18.436873 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:18.434732 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:18.543216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:18.542703 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fa32e97-4a60-4890-ae3f-0a66123b81b9" path="/var/lib/kubelet/pods/9fa32e97-4a60-4890-ae3f-0a66123b81b9/volumes" Oct 30 02:25:18.581841 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:18.581800 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:19.736263 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:19.736206 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:21.376628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:21.376583 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:22.390077 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:22.390030 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:22.508197 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:22.508117 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:23.264913 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.264879 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:23.292632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.292601 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q"] Oct 30 02:25:23.293261 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.293246 5998 cpu_manager.go:414] "containerMap: removing container" podUID="fbb9e882-4da2-4472-95ae-84ac0d5d419e" containerName="revoke-keys" Oct 30 02:25:23.293333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.293261 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb9e882-4da2-4472-95ae-84ac0d5d419e" containerName="revoke-keys" Oct 30 02:25:23.293333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.293280 5998 cpu_manager.go:414] "containerMap: removing container" podUID="d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed" containerName="revoke-keys" Oct 30 02:25:23.293333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.293286 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed" containerName="revoke-keys" Oct 30 02:25:23.293433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.293377 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed" containerName="revoke-keys" Oct 30 02:25:23.293433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.293395 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb9e882-4da2-4472-95ae-84ac0d5d419e" containerName="revoke-keys" Oct 30 02:25:23.298279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.298261 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q" Oct 30 02:25:23.327747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.326050 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:25:23.327747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.326309 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:25:23.327747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.326387 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:25:23.327747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.326508 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:25:23.360122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.356373 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q"] Oct 30 02:25:23.483743 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.483705 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:23.500146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.500079 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c846c2-743f-49ca-88e8-49a4936bfe23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c81c1028-xnz6q\" (UID: \"a9c846c2-743f-49ca-88e8-49a4936bfe23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q" Oct 30 02:25:23.504146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.500419 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c846c2-743f-49ca-88e8-49a4936bfe23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c81c1028-xnz6q\" (UID: \"a9c846c2-743f-49ca-88e8-49a4936bfe23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q" Oct 30 02:25:23.504146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.500478 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x866\" (UniqueName: \"kubernetes.io/projected/a9c846c2-743f-49ca-88e8-49a4936bfe23-kube-api-access-2x866\") pod \"maas-api-revoke-sub-c81c1028-xnz6q\" (UID: \"a9c846c2-743f-49ca-88e8-49a4936bfe23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q" Oct 30 02:25:23.607161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.603450 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c846c2-743f-49ca-88e8-49a4936bfe23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c81c1028-xnz6q\" (UID: \"a9c846c2-743f-49ca-88e8-49a4936bfe23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q" Oct 30 02:25:23.607161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.603738 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c846c2-743f-49ca-88e8-49a4936bfe23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c81c1028-xnz6q\" (UID: \"a9c846c2-743f-49ca-88e8-49a4936bfe23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q" Oct 30 02:25:23.607161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.603789 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x866\" (UniqueName: \"kubernetes.io/projected/a9c846c2-743f-49ca-88e8-49a4936bfe23-kube-api-access-2x866\") pod \"maas-api-revoke-sub-c81c1028-xnz6q\" (UID: \"a9c846c2-743f-49ca-88e8-49a4936bfe23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q" Oct 30 02:25:23.607161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.605614 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c846c2-743f-49ca-88e8-49a4936bfe23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c81c1028-xnz6q\" (UID: \"a9c846c2-743f-49ca-88e8-49a4936bfe23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q" Oct 30 02:25:23.625155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.622962 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:23.647118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.646878 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x866\" (UniqueName: \"kubernetes.io/projected/a9c846c2-743f-49ca-88e8-49a4936bfe23-kube-api-access-2x866\") pod \"maas-api-revoke-sub-c81c1028-xnz6q\" (UID: \"a9c846c2-743f-49ca-88e8-49a4936bfe23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q" Oct 30 02:25:23.671112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.658142 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c846c2-743f-49ca-88e8-49a4936bfe23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c81c1028-xnz6q\" (UID: \"a9c846c2-743f-49ca-88e8-49a4936bfe23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q" Oct 30 02:25:23.692117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.687345 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q" Oct 30 02:25:23.944573 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:23.944528 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:24.036872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:24.036839 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q"] Oct 30 02:25:24.039720 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:25:24.039673 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c846c2_743f_49ca_88e8_49a4936bfe23.slice/crio-10edf5e921b88316452c600b1dd927b0074d639f99f223a9dfc092c0c774cb16 WatchSource:0}: Error finding container 10edf5e921b88316452c600b1dd927b0074d639f99f223a9dfc092c0c774cb16: Status 404 returned error can't find the container with id 10edf5e921b88316452c600b1dd927b0074d639f99f223a9dfc092c0c774cb16 Oct 30 02:25:24.486383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:24.486348 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q" event={"ID":"a9c846c2-743f-49ca-88e8-49a4936bfe23","Type":"ContainerStarted","Data":"10edf5e921b88316452c600b1dd927b0074d639f99f223a9dfc092c0c774cb16"} Oct 30 02:25:25.101200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:25.101138 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:25.237930 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:25:25.237884 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:25:25.495340 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:25.495300 5998 generic.go:356] "Generic (PLEG): container finished" podID="a9c846c2-743f-49ca-88e8-49a4936bfe23" containerID="cb8d69a52deb1cd91ccd64b79ed4ee287e9d5181718708f266d4236b26a4de31" exitCode=0 Oct 30 02:25:25.495749 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:25.495386 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q" event={"ID":"a9c846c2-743f-49ca-88e8-49a4936bfe23","Type":"ContainerDied","Data":"cb8d69a52deb1cd91ccd64b79ed4ee287e9d5181718708f266d4236b26a4de31"} Oct 30 02:25:25.496447 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:25.496422 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:26.345115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:26.335959 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:26.463037 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:26.462972 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:26.687290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:26.687260 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q" Oct 30 02:25:26.780127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:26.780064 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9c846c2-743f-49ca-88e8-49a4936bfe23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c846c2-743f-49ca-88e8-49a4936bfe23-maas-api-openshift-ca\") pod \"a9c846c2-743f-49ca-88e8-49a4936bfe23\" (UID: \"a9c846c2-743f-49ca-88e8-49a4936bfe23\") " Oct 30 02:25:26.780335 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:26.780193 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9c846c2-743f-49ca-88e8-49a4936bfe23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c846c2-743f-49ca-88e8-49a4936bfe23-maas-api-xks-ca\") pod \"a9c846c2-743f-49ca-88e8-49a4936bfe23\" (UID: \"a9c846c2-743f-49ca-88e8-49a4936bfe23\") " Oct 30 02:25:26.780335 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:26.780252 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c846c2-743f-49ca-88e8-49a4936bfe23-kube-api-access-2x866\" (UniqueName: \"kubernetes.io/projected/a9c846c2-743f-49ca-88e8-49a4936bfe23-kube-api-access-2x866\") pod \"a9c846c2-743f-49ca-88e8-49a4936bfe23\" (UID: \"a9c846c2-743f-49ca-88e8-49a4936bfe23\") " Oct 30 02:25:26.780466 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:26.780415 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c846c2-743f-49ca-88e8-49a4936bfe23-maas-api-openshift-ca" pod "a9c846c2-743f-49ca-88e8-49a4936bfe23" (UID: "a9c846c2-743f-49ca-88e8-49a4936bfe23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:25:26.781061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:26.781032 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c846c2-743f-49ca-88e8-49a4936bfe23-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:25:26.782482 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:26.782452 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9c846c2-743f-49ca-88e8-49a4936bfe23-maas-api-xks-ca" pod "a9c846c2-743f-49ca-88e8-49a4936bfe23" (UID: "a9c846c2-743f-49ca-88e8-49a4936bfe23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:25:26.783075 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:26.783051 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c846c2-743f-49ca-88e8-49a4936bfe23-kube-api-access-2x866" pod "a9c846c2-743f-49ca-88e8-49a4936bfe23" (UID: "a9c846c2-743f-49ca-88e8-49a4936bfe23"). InnerVolumeSpecName "kube-api-access-2x866". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:25:26.882325 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:26.882286 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x866\" (UniqueName: \"kubernetes.io/projected/a9c846c2-743f-49ca-88e8-49a4936bfe23-kube-api-access-2x866\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:25:26.882325 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:26.882328 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c846c2-743f-49ca-88e8-49a4936bfe23-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:25:26.995444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:26.995402 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:27.512712 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:27.512680 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q" Oct 30 02:25:27.512906 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:27.512719 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q" event={"ID":"a9c846c2-743f-49ca-88e8-49a4936bfe23","Type":"ContainerDied","Data":"10edf5e921b88316452c600b1dd927b0074d639f99f223a9dfc092c0c774cb16"} Oct 30 02:25:27.512906 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:27.512772 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10edf5e921b88316452c600b1dd927b0074d639f99f223a9dfc092c0c774cb16" Oct 30 02:25:28.645674 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:28.645642 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:30.412155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:30.412118 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:30.590282 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:30.590249 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:32.165338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:32.165298 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:32.416370 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:32.416281 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:33.493476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:33.493443 5998 scope.go:122] "RemoveContainer" containerID="4509bd21b5b7eec3f48cb5fb12427b795f57049b9e3f806d5ae34ba74a7166c2" Oct 30 02:25:33.504579 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:33.504550 5998 scope.go:122] "RemoveContainer" containerID="d778228816e74ab67a4b0fdfd521b4ae0a5673179c0219e57f2a31932cf96f3f" Oct 30 02:25:33.812436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:33.812344 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr"] Oct 30 02:25:34.185673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:34.185642 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:35.086600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:35.086562 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:35.246117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:35.245194 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:36.487842 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:36.486375 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:36.727666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:36.727628 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:36.850723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:36.850631 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:37.091148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:37.085976 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:37.320575 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:37.320539 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:37.913431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:37.913393 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq"] Oct 30 02:25:37.975989 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:37.975958 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:38.380329 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:38.374676 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:38.813535 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:38.813487 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" podUID="f2088c14-033f-4f6c-86a4-bf49540d3d67" containerName="payload-processing" containerID="cri-o://d8f076ce0bdf1e82c2c7f3ecedd180c975d76b81608ca68eef621b8ff25580bd" gracePeriod=25 Oct 30 02:25:38.877417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:38.877385 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:39.118193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.118157 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" Oct 30 02:25:39.279550 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.279520 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2088c14-033f-4f6c-86a4-bf49540d3d67-kube-api-access-zzm5b\" (UniqueName: \"kubernetes.io/projected/f2088c14-033f-4f6c-86a4-bf49540d3d67-kube-api-access-zzm5b\") pod \"f2088c14-033f-4f6c-86a4-bf49540d3d67\" (UID: \"f2088c14-033f-4f6c-86a4-bf49540d3d67\") " Oct 30 02:25:39.279758 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.279570 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2088c14-033f-4f6c-86a4-bf49540d3d67-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2088c14-033f-4f6c-86a4-bf49540d3d67-routing-overlay-volume\") pod \"f2088c14-033f-4f6c-86a4-bf49540d3d67\" (UID: \"f2088c14-033f-4f6c-86a4-bf49540d3d67\") " Oct 30 02:25:39.279758 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.279634 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2088c14-033f-4f6c-86a4-bf49540d3d67-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2088c14-033f-4f6c-86a4-bf49540d3d67-provider-credentials\") pod \"f2088c14-033f-4f6c-86a4-bf49540d3d67\" (UID: \"f2088c14-033f-4f6c-86a4-bf49540d3d67\") " Oct 30 02:25:39.279758 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.279702 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2088c14-033f-4f6c-86a4-bf49540d3d67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2088c14-033f-4f6c-86a4-bf49540d3d67-plugins-config-volume\") pod \"f2088c14-033f-4f6c-86a4-bf49540d3d67\" (UID: \"f2088c14-033f-4f6c-86a4-bf49540d3d67\") " Oct 30 02:25:39.280022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.279988 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2088c14-033f-4f6c-86a4-bf49540d3d67-routing-overlay-volume" pod "f2088c14-033f-4f6c-86a4-bf49540d3d67" (UID: "f2088c14-033f-4f6c-86a4-bf49540d3d67"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:25:39.280189 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.280167 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2088c14-033f-4f6c-86a4-bf49540d3d67-plugins-config-volume" pod "f2088c14-033f-4f6c-86a4-bf49540d3d67" (UID: "f2088c14-033f-4f6c-86a4-bf49540d3d67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:25:39.280301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.280286 5998 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2088c14-033f-4f6c-86a4-bf49540d3d67-routing-overlay-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:25:39.280353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.280304 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2088c14-033f-4f6c-86a4-bf49540d3d67-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:25:39.281762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.281738 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2088c14-033f-4f6c-86a4-bf49540d3d67-kube-api-access-zzm5b" pod "f2088c14-033f-4f6c-86a4-bf49540d3d67" (UID: "f2088c14-033f-4f6c-86a4-bf49540d3d67"). InnerVolumeSpecName "kube-api-access-zzm5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:25:39.281859 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.281818 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2088c14-033f-4f6c-86a4-bf49540d3d67-provider-credentials" pod "f2088c14-033f-4f6c-86a4-bf49540d3d67" (UID: "f2088c14-033f-4f6c-86a4-bf49540d3d67"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:25:39.382385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.382293 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzm5b\" (UniqueName: \"kubernetes.io/projected/f2088c14-033f-4f6c-86a4-bf49540d3d67-kube-api-access-zzm5b\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:25:39.382385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.382329 5998 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2088c14-033f-4f6c-86a4-bf49540d3d67-provider-credentials\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:25:39.757760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.757717 5998 generic.go:356] "Generic (PLEG): container finished" podID="f2088c14-033f-4f6c-86a4-bf49540d3d67" containerID="d8f076ce0bdf1e82c2c7f3ecedd180c975d76b81608ca68eef621b8ff25580bd" exitCode=0 Oct 30 02:25:39.757974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.757763 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" Oct 30 02:25:39.757974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.757800 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" event={"ID":"f2088c14-033f-4f6c-86a4-bf49540d3d67","Type":"ContainerDied","Data":"d8f076ce0bdf1e82c2c7f3ecedd180c975d76b81608ca68eef621b8ff25580bd"} Oct 30 02:25:39.757974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.757857 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr" event={"ID":"f2088c14-033f-4f6c-86a4-bf49540d3d67","Type":"ContainerDied","Data":"7a944a7bdfe179d6545bf41b0c9c1737ddbff50dada9c80430064ce52060c5f6"} Oct 30 02:25:39.757974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.757882 5998 scope.go:122] "RemoveContainer" containerID="d8f076ce0bdf1e82c2c7f3ecedd180c975d76b81608ca68eef621b8ff25580bd" Oct 30 02:25:39.771541 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.771520 5998 scope.go:122] "RemoveContainer" containerID="d8f076ce0bdf1e82c2c7f3ecedd180c975d76b81608ca68eef621b8ff25580bd" Oct 30 02:25:39.771857 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:25:39.771831 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8f076ce0bdf1e82c2c7f3ecedd180c975d76b81608ca68eef621b8ff25580bd\": container with ID starting with d8f076ce0bdf1e82c2c7f3ecedd180c975d76b81608ca68eef621b8ff25580bd not found: ID does not exist" containerID="d8f076ce0bdf1e82c2c7f3ecedd180c975d76b81608ca68eef621b8ff25580bd" Oct 30 02:25:39.771935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.771870 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f076ce0bdf1e82c2c7f3ecedd180c975d76b81608ca68eef621b8ff25580bd"} err="failed to get container status \"d8f076ce0bdf1e82c2c7f3ecedd180c975d76b81608ca68eef621b8ff25580bd\": rpc error: code = NotFound desc = could not find container \"d8f076ce0bdf1e82c2c7f3ecedd180c975d76b81608ca68eef621b8ff25580bd\": container with ID starting with d8f076ce0bdf1e82c2c7f3ecedd180c975d76b81608ca68eef621b8ff25580bd not found: ID does not exist" Oct 30 02:25:39.806570 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.806540 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr"] Oct 30 02:25:39.819021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.818989 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-0ce950/payload-processing-external-model-e2e-extmod-a-0ce950-5d649rkgr"] Oct 30 02:25:39.947190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:39.947156 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:40.539952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:40.539913 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2088c14-033f-4f6c-86a4-bf49540d3d67" path="/var/lib/kubelet/pods/f2088c14-033f-4f6c-86a4-bf49540d3d67/volumes" Oct 30 02:25:41.576399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:41.576353 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:42.914802 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:42.914740 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" podUID="0fc698dd-2c33-4eaa-868d-9ebd6a706f4f" containerName="payload-processing" containerID="cri-o://95e746b8299ff5ec0b8783aa4b1d7a3017dc91de1e4d9b37525335b78cc3881c" gracePeriod=25 Oct 30 02:25:43.197305 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.197278 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" Oct 30 02:25:43.240039 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.239997 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-provider-credentials\") pod \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\" (UID: \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\") " Oct 30 02:25:43.241811 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.241785 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-routing-overlay-volume\") pod \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\" (UID: \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\") " Oct 30 02:25:43.242908 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.242320 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-routing-overlay-volume" pod "0fc698dd-2c33-4eaa-868d-9ebd6a706f4f" (UID: "0fc698dd-2c33-4eaa-868d-9ebd6a706f4f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:25:43.242908 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.242846 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-provider-credentials" pod "0fc698dd-2c33-4eaa-868d-9ebd6a706f4f" (UID: "0fc698dd-2c33-4eaa-868d-9ebd6a706f4f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:25:43.245318 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.245292 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-kube-api-access-gr5q9\" (UniqueName: \"kubernetes.io/projected/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-kube-api-access-gr5q9\") pod \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\" (UID: \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\") " Oct 30 02:25:43.245516 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.245488 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-plugins-config-volume\") pod \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\" (UID: \"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f\") " Oct 30 02:25:43.246199 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.246109 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-plugins-config-volume" pod "0fc698dd-2c33-4eaa-868d-9ebd6a706f4f" (UID: "0fc698dd-2c33-4eaa-868d-9ebd6a706f4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:25:43.247253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.247207 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:25:43.247404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.247381 5998 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-provider-credentials\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:25:43.247515 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.247492 5998 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-routing-overlay-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:25:43.248566 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.248544 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-kube-api-access-gr5q9" pod "0fc698dd-2c33-4eaa-868d-9ebd6a706f4f" (UID: "0fc698dd-2c33-4eaa-868d-9ebd6a706f4f"). InnerVolumeSpecName "kube-api-access-gr5q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:25:43.302547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.302518 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:43.358077 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.357444 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr5q9\" (UniqueName: \"kubernetes.io/projected/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f-kube-api-access-gr5q9\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:25:43.798423 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.798381 5998 generic.go:356] "Generic (PLEG): container finished" podID="0fc698dd-2c33-4eaa-868d-9ebd6a706f4f" containerID="95e746b8299ff5ec0b8783aa4b1d7a3017dc91de1e4d9b37525335b78cc3881c" exitCode=0 Oct 30 02:25:43.798592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.798447 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" Oct 30 02:25:43.798592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.798458 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" event={"ID":"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f","Type":"ContainerDied","Data":"95e746b8299ff5ec0b8783aa4b1d7a3017dc91de1e4d9b37525335b78cc3881c"} Oct 30 02:25:43.798592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.798520 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq" event={"ID":"0fc698dd-2c33-4eaa-868d-9ebd6a706f4f","Type":"ContainerDied","Data":"974e8c10685d4ef8346bef1d1eab0bef23bb97382336572a896051eeeaaea815"} Oct 30 02:25:43.798592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.798547 5998 scope.go:122] "RemoveContainer" containerID="95e746b8299ff5ec0b8783aa4b1d7a3017dc91de1e4d9b37525335b78cc3881c" Oct 30 02:25:43.813780 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.813756 5998 scope.go:122] "RemoveContainer" containerID="95e746b8299ff5ec0b8783aa4b1d7a3017dc91de1e4d9b37525335b78cc3881c" Oct 30 02:25:43.814112 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:25:43.814068 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e746b8299ff5ec0b8783aa4b1d7a3017dc91de1e4d9b37525335b78cc3881c\": container with ID starting with 95e746b8299ff5ec0b8783aa4b1d7a3017dc91de1e4d9b37525335b78cc3881c not found: ID does not exist" containerID="95e746b8299ff5ec0b8783aa4b1d7a3017dc91de1e4d9b37525335b78cc3881c" Oct 30 02:25:43.814208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.814142 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e746b8299ff5ec0b8783aa4b1d7a3017dc91de1e4d9b37525335b78cc3881c"} err="failed to get container status \"95e746b8299ff5ec0b8783aa4b1d7a3017dc91de1e4d9b37525335b78cc3881c\": rpc error: code = NotFound desc = could not find container \"95e746b8299ff5ec0b8783aa4b1d7a3017dc91de1e4d9b37525335b78cc3881c\": container with ID starting with 95e746b8299ff5ec0b8783aa4b1d7a3017dc91de1e4d9b37525335b78cc3881c not found: ID does not exist" Oct 30 02:25:43.849878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.849840 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq"] Oct 30 02:25:43.857212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:43.857186 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-93ec46/payload-processing-external-model-e2e-extmod-b-93ec46-7875jmkjq"] Oct 30 02:25:44.399177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:44.399142 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:44.525961 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:44.525922 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:44.541866 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:44.541828 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc698dd-2c33-4eaa-868d-9ebd6a706f4f" path="/var/lib/kubelet/pods/0fc698dd-2c33-4eaa-868d-9ebd6a706f4f/volumes" Oct 30 02:25:44.874774 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:44.874742 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:44.992386 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:44.992348 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:45.551372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.548520 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq"] Oct 30 02:25:45.551372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.549483 5998 cpu_manager.go:414] "containerMap: removing container" podUID="f2088c14-033f-4f6c-86a4-bf49540d3d67" containerName="payload-processing" Oct 30 02:25:45.551372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.549504 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2088c14-033f-4f6c-86a4-bf49540d3d67" containerName="payload-processing" Oct 30 02:25:45.551372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.549536 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a9c846c2-743f-49ca-88e8-49a4936bfe23" containerName="revoke-keys" Oct 30 02:25:45.551372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.549545 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c846c2-743f-49ca-88e8-49a4936bfe23" containerName="revoke-keys" Oct 30 02:25:45.551372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.549570 5998 cpu_manager.go:414] "containerMap: removing container" podUID="0fc698dd-2c33-4eaa-868d-9ebd6a706f4f" containerName="payload-processing" Oct 30 02:25:45.551372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.549578 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc698dd-2c33-4eaa-868d-9ebd6a706f4f" containerName="payload-processing" Oct 30 02:25:45.551372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.549783 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2088c14-033f-4f6c-86a4-bf49540d3d67" containerName="payload-processing" Oct 30 02:25:45.551372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.549800 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c846c2-743f-49ca-88e8-49a4936bfe23" containerName="revoke-keys" Oct 30 02:25:45.551372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.549814 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc698dd-2c33-4eaa-868d-9ebd6a706f4f" containerName="payload-processing" Oct 30 02:25:45.606745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.606654 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:45.689314 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.688292 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq"] Oct 30 02:25:45.689314 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.688633 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.704021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.703664 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5kqm\" (UniqueName: \"kubernetes.io/projected/2fd2791e-dcdd-40a2-be03-95c5c266d99c-kube-api-access-r5kqm\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.704021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.703711 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-workload-socket\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.704021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.703753 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-token\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.704021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.703778 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-workload-certs\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.704021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.703792 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istiod-ca-cert\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.704021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.703900 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-credential-socket\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.704021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.703915 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-envoy\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.704021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.703932 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-podinfo\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.704021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.704016 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-data\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.704622 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.704398 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-db01c4-openshift-default-dockercfg-9v57j\"" Oct 30 02:25:45.717110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.716266 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:45.809447 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.809358 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-data\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.809821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.809794 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5kqm\" (UniqueName: \"kubernetes.io/projected/2fd2791e-dcdd-40a2-be03-95c5c266d99c-kube-api-access-r5kqm\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.809984 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.809969 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-workload-socket\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.810149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.810135 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-token\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.810285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.810271 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-workload-certs\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.810487 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.810470 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istiod-ca-cert\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.810796 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.810778 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-credential-socket\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.810908 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.810894 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-envoy\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.811003 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.810990 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-podinfo\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.811834 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.811808 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-data\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.816307 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.815241 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-workload-certs\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.816508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.815818 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-workload-socket\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.816734 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.816348 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-credential-socket\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.817008 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.816980 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istiod-ca-cert\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.850306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.846908 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-envoy\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.850306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.847949 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-podinfo\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.850306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.848173 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5kqm\" (UniqueName: \"kubernetes.io/projected/2fd2791e-dcdd-40a2-be03-95c5c266d99c-kube-api-access-r5kqm\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:45.850306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:45.850244 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-token\") pod \"e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:46.009100 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:46.009038 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:46.355772 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:46.355745 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq"] Oct 30 02:25:46.358278 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:25:46.358252 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fd2791e_dcdd_40a2_be03_95c5c266d99c.slice/crio-1c9f2909b541414bd7a89fcc09ba180d068a9868cec524ea4b972cc7dea468d5 WatchSource:0}: Error finding container 1c9f2909b541414bd7a89fcc09ba180d068a9868cec524ea4b972cc7dea468d5: Status 404 returned error can't find the container with id 1c9f2909b541414bd7a89fcc09ba180d068a9868cec524ea4b972cc7dea468d5 Oct 30 02:25:46.361919 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:46.361886 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:25:46.362000 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:46.361952 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:25:46.362045 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:46.362002 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:25:46.633672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:46.633570 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:46.811014 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:46.810941 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:46.857151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:46.857109 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" event={"ID":"2fd2791e-dcdd-40a2-be03-95c5c266d99c","Type":"ContainerStarted","Data":"deaf88684552f21e3be7d70c5b86b55003114083a61b7d565382df82428faad2"} Oct 30 02:25:46.857312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:46.857173 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" event={"ID":"2fd2791e-dcdd-40a2-be03-95c5c266d99c","Type":"ContainerStarted","Data":"1c9f2909b541414bd7a89fcc09ba180d068a9868cec524ea4b972cc7dea468d5"} Oct 30 02:25:46.916274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:46.916208 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" podStartSLOduration=1.916188027 podStartE2EDuration="1.916188027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:25:46.910143644 +0000 UTC m=+3376.488934718" watchObservedRunningTime="2026-10-30 02:25:46.916188027 +0000 UTC m=+3376.494979110" Oct 30 02:25:46.949857 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:46.949819 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:47.009865 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:47.009824 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:47.011678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:47.011644 5998 patch_prober.go:29] interesting pod/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 02:25:47.011817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:47.011711 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" podUID="2fd2791e-dcdd-40a2-be03-95c5c266d99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 02:25:48.015980 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:48.015946 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:48.348309 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:48.348227 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:48.872380 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:48.872344 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:48.873365 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:48.873342 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:25:49.276790 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:49.276756 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:49.400079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:49.400048 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:50.131667 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:50.131614 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:25:50.245590 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:50.245560 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:51.100122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:51.074997 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:51.227271 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:51.227222 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:52.409710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:52.409663 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:54.217241 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:54.217188 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:25:54.249243 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:54.249187 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:25:54.263248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:54.263205 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 02:25:54.359239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:54.359184 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:25:54.394235 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:54.394172 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:25:54.502878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:54.502725 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:55.238592 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:25:55.238538 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:25:56.033592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:56.033555 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:57.587359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:57.587325 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:25:59.253242 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:25:59.253184 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:00.803306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:00.803246 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:02.466789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:02.466749 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:04.029842 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:04.029808 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:04.298463 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:04.298367 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:05.973063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:05.973025 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:07.548588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:07.548545 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:07.673971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:07.673931 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:09.263353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:09.263320 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:10.876627 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:10.876595 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:11.038434 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:11.038392 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2"] Oct 30 02:26:11.043672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:11.043647 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2"] Oct 30 02:26:11.050859 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:11.050834 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-374ef512-7rwv2"] Oct 30 02:26:11.054421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:11.054391 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e5090b-fwfs2"] Oct 30 02:26:12.540479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:12.540436 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ebf1325-84b3-47d8-a096-af69c5b55aec" path="/var/lib/kubelet/pods/4ebf1325-84b3-47d8-a096-af69c5b55aec/volumes" Oct 30 02:26:12.541155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:12.541131 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d703a3a6-1722-48d3-bbee-5913ccd2f487" path="/var/lib/kubelet/pods/d703a3a6-1722-48d3-bbee-5913ccd2f487/volumes" Oct 30 02:26:12.708259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:12.708203 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:14.340022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:14.339960 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:15.985481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:15.985446 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:17.692880 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:17.692830 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:19.286536 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:19.286505 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:20.945980 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:20.945940 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:21.053167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:21.053072 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:21.164955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:21.161771 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:22.700320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:22.700279 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:24.391823 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:24.391755 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:25.218665 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:26:25.218627 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:26:26.216869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:26.216828 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:28.210114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:28.204641 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:29.794564 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:29.794532 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:31.377507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:31.377420 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:32.999515 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:32.999488 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:33.549280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:33.549194 5998 scope.go:122] "RemoveContainer" containerID="6b4a9d0e7e8d93386f929d99d6e5fd857e8e112c4a4864753552986467b64668" Oct 30 02:26:33.561147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:33.561122 5998 scope.go:122] "RemoveContainer" containerID="5b8eb5f2e71f7f87cd0f399867a0422d83b4dde6905ed2f47fe2215695d939c4" Oct 30 02:26:33.571615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:33.571590 5998 scope.go:122] "RemoveContainer" containerID="936543a0b463010a691b88c9797ddfc3f8c274ac45fa7d3468bd18b5c6888cbf" Oct 30 02:26:33.582945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:33.582920 5998 scope.go:122] "RemoveContainer" containerID="88bd67b836c034b974a0f3b5ca0901994a64142fdf8d717aabfb64913c80a487" Oct 30 02:26:34.604300 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:34.604258 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:36.263607 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:36.263531 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:37.573883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:37.573848 5998 generic.go:356] "Generic (PLEG): container finished" podID="3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" containerID="2d7ea69ae41a5967fd68e4dc2bc2f12c95f6a766bd7de576d29f7b03492bbe5e" exitCode=0 Oct 30 02:26:37.574382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:37.573922 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" event={"ID":"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b","Type":"ContainerDied","Data":"2d7ea69ae41a5967fd68e4dc2bc2f12c95f6a766bd7de576d29f7b03492bbe5e"} Oct 30 02:26:37.574462 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:37.574444 5998 scope.go:122] "RemoveContainer" containerID="2d7ea69ae41a5967fd68e4dc2bc2f12c95f6a766bd7de576d29f7b03492bbe5e" Oct 30 02:26:37.576529 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:37.576501 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:26:37.576599 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:37.576573 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:26:37.576638 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:37.576609 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:26:37.856202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:37.855143 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:38.584615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:38.584580 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" event={"ID":"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b","Type":"ContainerStarted","Data":"6b484ea0b153eeb2afe624c009a9e4151c5563e55616c9a89173992d840e1a45"} Oct 30 02:26:38.998241 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:38.998198 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:39.204550 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:39.204518 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:26:39.204905 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:39.204883 5998 patch_prober.go:29] interesting pod/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 30 02:26:39.205075 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:39.205051 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" podUID="3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 30 02:26:39.459341 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:39.459313 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:39.570951 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:39.570880 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:39.681822 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:39.681788 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:40.206112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:40.205499 5998 patch_prober.go:29] interesting pod/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 30 02:26:40.206112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:40.205562 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" podUID="3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 30 02:26:41.170776 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:41.170724 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:41.230116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:41.224825 5998 patch_prober.go:29] interesting pod/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 30 02:26:41.230116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:41.224890 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" podUID="3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 30 02:26:42.237369 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:42.237257 5998 patch_prober.go:29] interesting pod/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 30 02:26:42.237877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:42.237366 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" podUID="3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 30 02:26:42.679537 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:42.679501 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:43.211117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:43.210642 5998 patch_prober.go:29] interesting pod/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 30 02:26:43.211117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:43.210722 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" podUID="3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 30 02:26:43.623173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:43.621063 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:43.732653 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:43.732610 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:44.222378 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:44.222342 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:26:44.223412 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:44.223387 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:26:44.224940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:44.224911 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:26:44.291418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:44.291374 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:45.954927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:45.954891 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:47.496941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:47.496902 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:49.180921 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:49.180885 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:50.781550 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:50.781493 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:52.346857 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:52.346820 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:53.933056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:53.933021 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:55.216973 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:26:55.216927 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:26:55.596576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:55.596492 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:55.702801 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:55.702767 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:57.308304 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:57.308267 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:26:59.160352 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:26:59.160310 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:00.963673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:00.963629 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:01.757973 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:01.757938 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:02.187149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.187116 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:02.589108 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.589011 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m"] Oct 30 02:27:02.593241 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.593216 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.621285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.616664 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4593af-openshift-default-dockercfg-cnzsh\"" Oct 30 02:27:02.668858 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.668769 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:02.673864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.673831 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-workload-socket\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.677450 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.675441 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m"] Oct 30 02:27:02.799114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.795453 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-podinfo\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.881196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.875546 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025cc8ef-572d-4f6a-851c-2e59fb797189-istiod-ca-cert\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.881196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.871046 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:02.881196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.875841 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-token\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.881196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.875931 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-envoy\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.881196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.876138 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-workload-certs\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.881196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.876205 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2htds\" (UniqueName: \"kubernetes.io/projected/025cc8ef-572d-4f6a-851c-2e59fb797189-kube-api-access-2htds\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.881196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.876236 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-data\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.881196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.876449 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-credential-socket\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.881196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.876503 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-workload-socket\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.881196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.877209 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-workload-socket\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.979108 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.979047 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-envoy\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.979489 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.979466 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-workload-certs\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.979642 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.979627 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2htds\" (UniqueName: \"kubernetes.io/projected/025cc8ef-572d-4f6a-851c-2e59fb797189-kube-api-access-2htds\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.979759 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.979744 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-data\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.979977 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.979960 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-credential-socket\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.980187 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.980168 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-podinfo\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.992153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.980360 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025cc8ef-572d-4f6a-851c-2e59fb797189-istiod-ca-cert\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.992504 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.992473 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-token\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:02.993811 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.993781 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-data\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:03.016427 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.994245 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-workload-certs\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:03.016693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.995324 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025cc8ef-572d-4f6a-851c-2e59fb797189-istiod-ca-cert\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:03.016837 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.995844 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-credential-socket\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:03.016959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:02.997905 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:03.066109 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:03.064841 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-envoy\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:03.067119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:03.066790 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-podinfo\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:03.111113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:03.110483 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2htds\" (UniqueName: \"kubernetes.io/projected/025cc8ef-572d-4f6a-851c-2e59fb797189-kube-api-access-2htds\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:03.133175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:03.119802 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-token\") pod \"e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:03.273457 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:03.273416 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:03.748738 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:03.748629 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m"] Oct 30 02:27:03.755995 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:27:03.755952 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025cc8ef_572d_4f6a_851c_2e59fb797189.slice/crio-adfe47df0e8760111d80357094b6478f280f4d8c080c82e4a226c00abe90c35d WatchSource:0}: Error finding container adfe47df0e8760111d80357094b6478f280f4d8c080c82e4a226c00abe90c35d: Status 404 returned error can't find the container with id adfe47df0e8760111d80357094b6478f280f4d8c080c82e4a226c00abe90c35d Oct 30 02:27:03.759824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:03.759778 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:27:03.759946 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:03.759879 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:27:03.759946 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:03.759921 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:27:03.965033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:03.964997 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" event={"ID":"025cc8ef-572d-4f6a-851c-2e59fb797189","Type":"ContainerStarted","Data":"1cc92474d04402c331a74bccbbd597eefb521f2411ec393cecf2a3348146d7d5"} Oct 30 02:27:03.965248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:03.965060 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" event={"ID":"025cc8ef-572d-4f6a-851c-2e59fb797189","Type":"ContainerStarted","Data":"adfe47df0e8760111d80357094b6478f280f4d8c080c82e4a226c00abe90c35d"} Oct 30 02:27:04.016178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:04.014110 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" podStartSLOduration=2.014071883 podStartE2EDuration="2.014071883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:27:04.012676795 +0000 UTC m=+3453.591467888" watchObservedRunningTime="2026-10-30 02:27:04.014071883 +0000 UTC m=+3453.592862965" Oct 30 02:27:04.226877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:04.226730 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:04.280312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:04.278775 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:04.281437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:04.281031 5998 patch_prober.go:29] interesting pod/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 02:27:04.281437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:04.281125 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" podUID="025cc8ef-572d-4f6a-851c-2e59fb797189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 02:27:05.016600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:05.016561 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:05.280546 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:05.280469 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:05.980587 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:05.980549 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:05.981895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:05.981868 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:27:06.074938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:06.074901 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:06.402759 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:06.402631 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:06.531323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:06.531288 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:07.650399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:07.650354 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:07.759050 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:07.759016 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:07.863624 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:07.863591 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj"] Oct 30 02:27:07.868930 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:07.868901 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:07.894140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:07.893807 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3d5da6\"/\"kube-root-ca.crt\"" Oct 30 02:27:07.930844 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:07.902793 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3d5da6\"/\"openshift-service-ca.crt\"" Oct 30 02:27:07.930844 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:07.909983 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj"] Oct 30 02:27:07.946198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:07.944729 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:07.953308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:07.953277 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3d5da6\"/\"test-model-3d5da6-kserve-self-signed-certs\"" Oct 30 02:27:07.953594 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:07.953576 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3d5da6\"/\"default-dockercfg-4kdrp\"" Oct 30 02:27:07.972058 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:07.972020 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfd89\" (UniqueName: \"kubernetes.io/projected/44d99268-92b1-4b8b-8061-4b5732b2de40-kube-api-access-bfd89\") pod \"test-model-3d5da6-kserve-846bd97fbf-qdpwj\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:07.972254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:07.972178 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-model-cache\") pod \"test-model-3d5da6-kserve-846bd97fbf-qdpwj\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:07.972353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:07.972329 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-tmp-dir\") pod \"test-model-3d5da6-kserve-846bd97fbf-qdpwj\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:07.972419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:07.972392 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-dshm\") pod \"test-model-3d5da6-kserve-846bd97fbf-qdpwj\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:07.972489 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:07.972466 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d99268-92b1-4b8b-8061-4b5732b2de40-tls-certs\") pod \"test-model-3d5da6-kserve-846bd97fbf-qdpwj\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:07.972554 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:07.972500 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-home\") pod \"test-model-3d5da6-kserve-846bd97fbf-qdpwj\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:08.064826 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:08.064793 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:08.075791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:08.075196 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d99268-92b1-4b8b-8061-4b5732b2de40-tls-certs\") pod \"test-model-3d5da6-kserve-846bd97fbf-qdpwj\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:08.075791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:08.075257 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-home\") pod \"test-model-3d5da6-kserve-846bd97fbf-qdpwj\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:08.075791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:08.075335 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfd89\" (UniqueName: \"kubernetes.io/projected/44d99268-92b1-4b8b-8061-4b5732b2de40-kube-api-access-bfd89\") pod \"test-model-3d5da6-kserve-846bd97fbf-qdpwj\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:08.075791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:08.075445 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-model-cache\") pod \"test-model-3d5da6-kserve-846bd97fbf-qdpwj\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:08.075791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:08.075628 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-tmp-dir\") pod \"test-model-3d5da6-kserve-846bd97fbf-qdpwj\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:08.075791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:08.075664 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-dshm\") pod \"test-model-3d5da6-kserve-846bd97fbf-qdpwj\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:08.076287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:08.075882 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-home\") pod \"test-model-3d5da6-kserve-846bd97fbf-qdpwj\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:08.076719 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:08.076617 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-tmp-dir\") pod \"test-model-3d5da6-kserve-846bd97fbf-qdpwj\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:08.079134 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:08.078484 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-model-cache\") pod \"test-model-3d5da6-kserve-846bd97fbf-qdpwj\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:08.080593 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:08.080505 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-dshm\") pod \"test-model-3d5da6-kserve-846bd97fbf-qdpwj\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:08.082548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:08.082524 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d99268-92b1-4b8b-8061-4b5732b2de40-tls-certs\") pod \"test-model-3d5da6-kserve-846bd97fbf-qdpwj\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:08.115233 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:08.115193 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfd89\" (UniqueName: \"kubernetes.io/projected/44d99268-92b1-4b8b-8061-4b5732b2de40-kube-api-access-bfd89\") pod \"test-model-3d5da6-kserve-846bd97fbf-qdpwj\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:08.178035 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:08.177979 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:08.184979 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:08.184683 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:08.295651 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:08.295614 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:08.794350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:08.793904 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj"] Oct 30 02:27:09.010323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:09.010227 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" event={"ID":"44d99268-92b1-4b8b-8061-4b5732b2de40","Type":"ContainerStarted","Data":"31e66cf31f29371c6a5628d93b23258af9effca76a8a93437648269a969a6982"} Oct 30 02:27:09.011072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:09.011036 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" event={"ID":"44d99268-92b1-4b8b-8061-4b5732b2de40","Type":"ContainerStarted","Data":"e73f947794c4e7b65dea1fea9e4f1b0678afb391e3345527c25d6a9726b21591"} Oct 30 02:27:09.012721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:09.012697 5998 generic.go:356] "Generic (PLEG): container finished" podID="bf65846d-6629-424c-9eda-4c359fd1992a" containerID="8ee63260b4416374d92f1a6eb0f58f9d8fb329fb304a04685a8d2622b454fca2" exitCode=0 Oct 30 02:27:09.012856 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:09.012738 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" event={"ID":"bf65846d-6629-424c-9eda-4c359fd1992a","Type":"ContainerDied","Data":"8ee63260b4416374d92f1a6eb0f58f9d8fb329fb304a04685a8d2622b454fca2"} Oct 30 02:27:09.013169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:09.013150 5998 scope.go:122] "RemoveContainer" containerID="8ee63260b4416374d92f1a6eb0f58f9d8fb329fb304a04685a8d2622b454fca2" Oct 30 02:27:09.015943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:09.015866 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:27:09.016052 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:09.015964 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:27:09.016052 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:09.016011 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:27:09.062377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:09.061822 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" podStartSLOduration=2.061803854 podStartE2EDuration="2.061803854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:27:09.058959497 +0000 UTC m=+3458.637750575" watchObservedRunningTime="2026-10-30 02:27:09.061803854 +0000 UTC m=+3458.640594936" Oct 30 02:27:09.185617 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:09.185593 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:09.229019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:09.228893 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:27:09.903185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:09.903147 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:10.027330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:10.027283 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" event={"ID":"bf65846d-6629-424c-9eda-4c359fd1992a","Type":"ContainerStarted","Data":"3c7d4f5f4edb8be2cc0aabd86366460c6945492f4a4e5471a0178e8b585da5fa"} Oct 30 02:27:10.918115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:10.903763 5998 patch_prober.go:29] interesting pod/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 30 02:27:10.918115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:10.903847 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" podUID="bf65846d-6629-424c-9eda-4c359fd1992a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 30 02:27:10.918115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:10.904700 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:27:11.516274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:11.514122 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:27:11.621181 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:11.621142 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:11.735699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:11.735647 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:11.874639 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:11.874550 5998 patch_prober.go:29] interesting pod/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 30 02:27:11.874909 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:11.874882 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" podUID="bf65846d-6629-424c-9eda-4c359fd1992a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 30 02:27:12.871556 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:12.871525 5998 patch_prober.go:29] interesting pod/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 30 02:27:12.872016 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:12.871990 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" podUID="bf65846d-6629-424c-9eda-4c359fd1992a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 30 02:27:13.113115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:13.112205 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:27:13.374279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:13.374248 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:13.873374 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:13.871485 5998 patch_prober.go:29] interesting pod/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 30 02:27:13.873374 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:13.871548 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" podUID="bf65846d-6629-424c-9eda-4c359fd1992a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 30 02:27:14.795188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:14.795155 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7"] Oct 30 02:27:14.799988 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:14.799969 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7" Oct 30 02:27:14.839509 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:14.838147 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:27:14.839509 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:14.838221 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:27:14.839509 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:14.838303 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:27:14.839509 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:14.838404 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:27:14.852120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:14.851822 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7"] Oct 30 02:27:14.953113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:14.952769 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3507eb0-qwlf7\" (UID: \"ae0f38fe-b272-4916-89a3-7dcf32ea18d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7" Oct 30 02:27:14.992113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:14.991731 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3507eb0-qwlf7\" (UID: \"ae0f38fe-b272-4916-89a3-7dcf32ea18d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7" Oct 30 02:27:14.992113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:14.991838 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlldf\" (UniqueName: \"kubernetes.io/projected/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-kube-api-access-vlldf\") pod \"maas-api-revoke-sub-d3507eb0-qwlf7\" (UID: \"ae0f38fe-b272-4916-89a3-7dcf32ea18d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7" Oct 30 02:27:15.035121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:15.034260 5998 patch_prober.go:29] interesting pod/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 30 02:27:15.089168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:15.088372 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" podUID="bf65846d-6629-424c-9eda-4c359fd1992a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 30 02:27:15.114119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:15.111795 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3507eb0-qwlf7\" (UID: \"ae0f38fe-b272-4916-89a3-7dcf32ea18d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7" Oct 30 02:27:15.114119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:15.111868 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlldf\" (UniqueName: \"kubernetes.io/projected/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-kube-api-access-vlldf\") pod \"maas-api-revoke-sub-d3507eb0-qwlf7\" (UID: \"ae0f38fe-b272-4916-89a3-7dcf32ea18d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7" Oct 30 02:27:15.114119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:15.112188 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3507eb0-qwlf7\" (UID: \"ae0f38fe-b272-4916-89a3-7dcf32ea18d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7" Oct 30 02:27:15.141115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:15.115157 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3507eb0-qwlf7\" (UID: \"ae0f38fe-b272-4916-89a3-7dcf32ea18d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7" Oct 30 02:27:15.164837 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:15.162634 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3507eb0-qwlf7\" (UID: \"ae0f38fe-b272-4916-89a3-7dcf32ea18d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7" Oct 30 02:27:15.190801 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:15.182943 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlldf\" (UniqueName: \"kubernetes.io/projected/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-kube-api-access-vlldf\") pod \"maas-api-revoke-sub-d3507eb0-qwlf7\" (UID: \"ae0f38fe-b272-4916-89a3-7dcf32ea18d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7" Oct 30 02:27:15.425106 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:15.424877 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:15.459490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:15.459457 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7" Oct 30 02:27:15.871914 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:15.871874 5998 patch_prober.go:29] interesting pod/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 30 02:27:15.872126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:15.871948 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" podUID="bf65846d-6629-424c-9eda-4c359fd1992a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 30 02:27:15.960334 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:15.960018 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7"] Oct 30 02:27:16.292291 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:16.292082 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7" event={"ID":"ae0f38fe-b272-4916-89a3-7dcf32ea18d6","Type":"ContainerStarted","Data":"d25a18da55a9a335539db2c98acf7b5d5156e1b20a0099fed277757e92220a2f"} Oct 30 02:27:16.872130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:16.872085 5998 patch_prober.go:29] interesting pod/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 30 02:27:16.872355 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:16.872187 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" podUID="bf65846d-6629-424c-9eda-4c359fd1992a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 30 02:27:17.147789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:17.147752 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7"] Oct 30 02:27:17.148371 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:17.147957 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" podUID="41479a83-7edc-4856-b62c-e12ee3c2d814" containerName="istio-proxy" containerID="cri-o://d64ee70885995a31b032b539791e26f2b8a07643aa3cce12f25b9723265588f0" gracePeriod=30 Oct 30 02:27:17.288630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:17.288592 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:17.337112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:17.336524 5998 generic.go:356] "Generic (PLEG): container finished" podID="ae0f38fe-b272-4916-89a3-7dcf32ea18d6" containerID="dcab84bc209875a52fced0017bc2b0c70016cd4e31c62f3014759c4759525e1f" exitCode=0 Oct 30 02:27:17.337112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:17.336600 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7" event={"ID":"ae0f38fe-b272-4916-89a3-7dcf32ea18d6","Type":"ContainerDied","Data":"dcab84bc209875a52fced0017bc2b0c70016cd4e31c62f3014759c4759525e1f"} Oct 30 02:27:17.876711 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:17.876679 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:27:17.877912 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:17.877883 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:27:17.887423 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:17.887397 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:18.185165 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:18.185128 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:18.199694 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:18.199660 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:18.350571 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:18.350539 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:18.356285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:18.356256 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:27:18.553242 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:18.553210 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7" Oct 30 02:27:18.625057 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:18.625020 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-maas-api-openshift-ca\") pod \"ae0f38fe-b272-4916-89a3-7dcf32ea18d6\" (UID: \"ae0f38fe-b272-4916-89a3-7dcf32ea18d6\") " Oct 30 02:27:18.625307 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:18.625190 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-maas-api-xks-ca\") pod \"ae0f38fe-b272-4916-89a3-7dcf32ea18d6\" (UID: \"ae0f38fe-b272-4916-89a3-7dcf32ea18d6\") " Oct 30 02:27:18.625307 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:18.625288 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-kube-api-access-vlldf\" (UniqueName: \"kubernetes.io/projected/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-kube-api-access-vlldf\") pod \"ae0f38fe-b272-4916-89a3-7dcf32ea18d6\" (UID: \"ae0f38fe-b272-4916-89a3-7dcf32ea18d6\") " Oct 30 02:27:18.625486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:18.625457 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-maas-api-openshift-ca" pod "ae0f38fe-b272-4916-89a3-7dcf32ea18d6" (UID: "ae0f38fe-b272-4916-89a3-7dcf32ea18d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:27:18.626339 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:18.626173 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:18.628631 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:18.628600 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-kube-api-access-vlldf" pod "ae0f38fe-b272-4916-89a3-7dcf32ea18d6" (UID: "ae0f38fe-b272-4916-89a3-7dcf32ea18d6"). InnerVolumeSpecName "kube-api-access-vlldf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:27:18.633228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:18.633201 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-maas-api-xks-ca" pod "ae0f38fe-b272-4916-89a3-7dcf32ea18d6" (UID: "ae0f38fe-b272-4916-89a3-7dcf32ea18d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:27:18.728006 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:18.727964 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlldf\" (UniqueName: \"kubernetes.io/projected/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-kube-api-access-vlldf\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:18.728006 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:18.727998 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae0f38fe-b272-4916-89a3-7dcf32ea18d6-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:18.923937 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:18.923839 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:18.998674 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:18.998300 5998 patch_prober.go:29] interesting pod/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:27:18.998674 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:18.998362 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" podUID="41479a83-7edc-4856-b62c-e12ee3c2d814" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:27:19.085032 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:19.084949 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:19.331505 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:19.331402 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:19.410627 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:19.410597 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7" Oct 30 02:27:19.410834 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:19.410678 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7" event={"ID":"ae0f38fe-b272-4916-89a3-7dcf32ea18d6","Type":"ContainerDied","Data":"d25a18da55a9a335539db2c98acf7b5d5156e1b20a0099fed277757e92220a2f"} Oct 30 02:27:19.410834 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:19.410734 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d25a18da55a9a335539db2c98acf7b5d5156e1b20a0099fed277757e92220a2f" Oct 30 02:27:19.924299 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:19.924251 5998 patch_prober.go:29] interesting pod/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:27:19.924505 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:19.924319 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" podUID="2fd2791e-dcdd-40a2-be03-95c5c266d99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:27:20.820326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:20.820288 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:22.443893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.443506 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 02:27:22.454132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.454085 5998 generic.go:356] "Generic (PLEG): container finished" podID="2fd2791e-dcdd-40a2-be03-95c5c266d99c" containerID="deaf88684552f21e3be7d70c5b86b55003114083a61b7d565382df82428faad2" exitCode=0 Oct 30 02:27:22.454311 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.454147 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" event={"ID":"2fd2791e-dcdd-40a2-be03-95c5c266d99c","Type":"ContainerDied","Data":"deaf88684552f21e3be7d70c5b86b55003114083a61b7d565382df82428faad2"} Oct 30 02:27:22.454707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.454687 5998 scope.go:122] "RemoveContainer" containerID="deaf88684552f21e3be7d70c5b86b55003114083a61b7d565382df82428faad2" Oct 30 02:27:22.455938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.455911 5998 generic.go:356] "Generic (PLEG): container finished" podID="41479a83-7edc-4856-b62c-e12ee3c2d814" containerID="d64ee70885995a31b032b539791e26f2b8a07643aa3cce12f25b9723265588f0" exitCode=0 Oct 30 02:27:22.456051 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.455976 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" Oct 30 02:27:22.456162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.456048 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" event={"ID":"41479a83-7edc-4856-b62c-e12ee3c2d814","Type":"ContainerDied","Data":"d64ee70885995a31b032b539791e26f2b8a07643aa3cce12f25b9723265588f0"} Oct 30 02:27:22.456162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.456127 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7" event={"ID":"41479a83-7edc-4856-b62c-e12ee3c2d814","Type":"ContainerDied","Data":"ddcc60c550a402178db2b8856e3984e073fcf1fa54331dcaa469b50c62de77fe"} Oct 30 02:27:22.456162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.456157 5998 scope.go:122] "RemoveContainer" containerID="d64ee70885995a31b032b539791e26f2b8a07643aa3cce12f25b9723265588f0" Oct 30 02:27:22.457039 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.457013 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:27:22.457184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.457100 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:27:22.457184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.457149 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:27:22.473168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.473145 5998 scope.go:122] "RemoveContainer" containerID="d64ee70885995a31b032b539791e26f2b8a07643aa3cce12f25b9723265588f0" Oct 30 02:27:22.473501 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:27:22.473481 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d64ee70885995a31b032b539791e26f2b8a07643aa3cce12f25b9723265588f0\": container with ID starting with d64ee70885995a31b032b539791e26f2b8a07643aa3cce12f25b9723265588f0 not found: ID does not exist" containerID="d64ee70885995a31b032b539791e26f2b8a07643aa3cce12f25b9723265588f0" Oct 30 02:27:22.473592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.473513 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64ee70885995a31b032b539791e26f2b8a07643aa3cce12f25b9723265588f0"} err="failed to get container status \"d64ee70885995a31b032b539791e26f2b8a07643aa3cce12f25b9723265588f0\": rpc error: code = NotFound desc = could not find container \"d64ee70885995a31b032b539791e26f2b8a07643aa3cce12f25b9723265588f0\": container with ID starting with d64ee70885995a31b032b539791e26f2b8a07643aa3cce12f25b9723265588f0 not found: ID does not exist" Oct 30 02:27:22.566253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.563544 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-data\") pod \"41479a83-7edc-4856-b62c-e12ee3c2d814\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " Oct 30 02:27:22.566253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.563611 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-token\" (UniqueName: \"kubernetes.io/projected/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-token\") pod \"41479a83-7edc-4856-b62c-e12ee3c2d814\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " Oct 30 02:27:22.566253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.563660 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-workload-certs\") pod \"41479a83-7edc-4856-b62c-e12ee3c2d814\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " Oct 30 02:27:22.566253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.563730 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-podinfo\") pod \"41479a83-7edc-4856-b62c-e12ee3c2d814\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " Oct 30 02:27:22.566253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.563826 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41479a83-7edc-4856-b62c-e12ee3c2d814-kube-api-access-qpkjp\" (UniqueName: \"kubernetes.io/projected/41479a83-7edc-4856-b62c-e12ee3c2d814-kube-api-access-qpkjp\") pod \"41479a83-7edc-4856-b62c-e12ee3c2d814\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " Oct 30 02:27:22.566253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.563881 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-envoy\") pod \"41479a83-7edc-4856-b62c-e12ee3c2d814\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " Oct 30 02:27:22.566253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.563958 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41479a83-7edc-4856-b62c-e12ee3c2d814-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41479a83-7edc-4856-b62c-e12ee3c2d814-istiod-ca-cert\") pod \"41479a83-7edc-4856-b62c-e12ee3c2d814\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " Oct 30 02:27:22.566253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.563997 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-workload-socket\") pod \"41479a83-7edc-4856-b62c-e12ee3c2d814\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " Oct 30 02:27:22.566253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.564129 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-credential-socket\") pod \"41479a83-7edc-4856-b62c-e12ee3c2d814\" (UID: \"41479a83-7edc-4856-b62c-e12ee3c2d814\") " Oct 30 02:27:22.566706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.566401 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-data" pod "41479a83-7edc-4856-b62c-e12ee3c2d814" (UID: "41479a83-7edc-4856-b62c-e12ee3c2d814"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:27:22.567710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.567649 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-workload-certs" pod "41479a83-7edc-4856-b62c-e12ee3c2d814" (UID: "41479a83-7edc-4856-b62c-e12ee3c2d814"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:27:22.568244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.568211 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-workload-socket" pod "41479a83-7edc-4856-b62c-e12ee3c2d814" (UID: "41479a83-7edc-4856-b62c-e12ee3c2d814"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:27:22.568409 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.568384 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41479a83-7edc-4856-b62c-e12ee3c2d814-istiod-ca-cert" pod "41479a83-7edc-4856-b62c-e12ee3c2d814" (UID: "41479a83-7edc-4856-b62c-e12ee3c2d814"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:27:22.568520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.568501 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-credential-socket" pod "41479a83-7edc-4856-b62c-e12ee3c2d814" (UID: "41479a83-7edc-4856-b62c-e12ee3c2d814"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:27:22.569180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.569154 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-envoy" pod "41479a83-7edc-4856-b62c-e12ee3c2d814" (UID: "41479a83-7edc-4856-b62c-e12ee3c2d814"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:27:22.569267 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.569165 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-podinfo" pod "41479a83-7edc-4856-b62c-e12ee3c2d814" (UID: "41479a83-7edc-4856-b62c-e12ee3c2d814"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:27:22.569338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.569310 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-token" pod "41479a83-7edc-4856-b62c-e12ee3c2d814" (UID: "41479a83-7edc-4856-b62c-e12ee3c2d814"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:27:22.570281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.570257 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41479a83-7edc-4856-b62c-e12ee3c2d814-kube-api-access-qpkjp" pod "41479a83-7edc-4856-b62c-e12ee3c2d814" (UID: "41479a83-7edc-4856-b62c-e12ee3c2d814"). InnerVolumeSpecName "kube-api-access-qpkjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:27:22.687326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.685798 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:22.687326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.685838 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:22.687326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.685851 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:22.687326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.685863 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:22.687326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.685875 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpkjp\" (UniqueName: \"kubernetes.io/projected/41479a83-7edc-4856-b62c-e12ee3c2d814-kube-api-access-qpkjp\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:22.687326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.685887 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:22.687326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.685899 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41479a83-7edc-4856-b62c-e12ee3c2d814-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:22.687326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.685910 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:22.687326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.685922 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41479a83-7edc-4856-b62c-e12ee3c2d814-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:22.815081 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.812451 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:22.863125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.863056 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7"] Oct 30 02:27:22.868044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.868016 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-5tfx7"] Oct 30 02:27:22.891571 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:27:22.891533 5998 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41479a83_7edc_4856_b62c_e12ee3c2d814.slice/crio-ddcc60c550a402178db2b8856e3984e073fcf1fa54331dcaa469b50c62de77fe\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41479a83_7edc_4856_b62c_e12ee3c2d814.slice\": RecentStats: unable to find data in memory cache]" Oct 30 02:27:22.947432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:22.947400 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:23.467046 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:23.466952 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" event={"ID":"2fd2791e-dcdd-40a2-be03-95c5c266d99c","Type":"ContainerStarted","Data":"31427e52ef612d3ab97df4e30432e98ecf649b426c1a43e9a6da808ae6a0a480"} Oct 30 02:27:24.030112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:24.016472 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:27:24.030112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:24.016642 5998 patch_prober.go:29] interesting pod/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 02:27:24.030112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:24.016688 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" podUID="2fd2791e-dcdd-40a2-be03-95c5c266d99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 02:27:24.437992 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:24.437519 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:24.540238 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:24.540201 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41479a83-7edc-4856-b62c-e12ee3c2d814" path="/var/lib/kubelet/pods/41479a83-7edc-4856-b62c-e12ee3c2d814/volumes" Oct 30 02:27:24.544069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:24.544016 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:25.058146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.054242 5998 patch_prober.go:29] interesting pod/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 02:27:25.058146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.054311 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" podUID="2fd2791e-dcdd-40a2-be03-95c5c266d99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 02:27:25.250919 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:27:25.250881 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:27:25.555276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.555238 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9"] Oct 30 02:27:25.555869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.555852 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ae0f38fe-b272-4916-89a3-7dcf32ea18d6" containerName="revoke-keys" Oct 30 02:27:25.555869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.555867 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae0f38fe-b272-4916-89a3-7dcf32ea18d6" containerName="revoke-keys" Oct 30 02:27:25.555992 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.555899 5998 cpu_manager.go:414] "containerMap: removing container" podUID="41479a83-7edc-4856-b62c-e12ee3c2d814" containerName="istio-proxy" Oct 30 02:27:25.555992 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.555905 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41479a83-7edc-4856-b62c-e12ee3c2d814" containerName="istio-proxy" Oct 30 02:27:25.556112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.556003 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="41479a83-7edc-4856-b62c-e12ee3c2d814" containerName="istio-proxy" Oct 30 02:27:25.556112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.556017 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae0f38fe-b272-4916-89a3-7dcf32ea18d6" containerName="revoke-keys" Oct 30 02:27:25.561240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.561216 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9" Oct 30 02:27:25.582072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.582046 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9"] Oct 30 02:27:25.586727 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.586701 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:27:25.586727 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.586701 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:27:25.587026 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.587002 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:27:25.591354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.590279 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:27:25.739178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.739136 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429cee8e-bd15-4c14-800d-8e48377c9ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d91bc928-pxfr9\" (UID: \"429cee8e-bd15-4c14-800d-8e48377c9ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9" Oct 30 02:27:25.739374 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.739189 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nprxz\" (UniqueName: \"kubernetes.io/projected/429cee8e-bd15-4c14-800d-8e48377c9ec2-kube-api-access-nprxz\") pod \"maas-api-revoke-sub-d91bc928-pxfr9\" (UID: \"429cee8e-bd15-4c14-800d-8e48377c9ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9" Oct 30 02:27:25.739374 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.739365 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429cee8e-bd15-4c14-800d-8e48377c9ec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d91bc928-pxfr9\" (UID: \"429cee8e-bd15-4c14-800d-8e48377c9ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9" Oct 30 02:27:25.840613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.840511 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429cee8e-bd15-4c14-800d-8e48377c9ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d91bc928-pxfr9\" (UID: \"429cee8e-bd15-4c14-800d-8e48377c9ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9" Oct 30 02:27:25.840613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.840561 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nprxz\" (UniqueName: \"kubernetes.io/projected/429cee8e-bd15-4c14-800d-8e48377c9ec2-kube-api-access-nprxz\") pod \"maas-api-revoke-sub-d91bc928-pxfr9\" (UID: \"429cee8e-bd15-4c14-800d-8e48377c9ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9" Oct 30 02:27:25.840863 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.840656 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429cee8e-bd15-4c14-800d-8e48377c9ec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d91bc928-pxfr9\" (UID: \"429cee8e-bd15-4c14-800d-8e48377c9ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9" Oct 30 02:27:25.841313 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.841289 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429cee8e-bd15-4c14-800d-8e48377c9ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d91bc928-pxfr9\" (UID: \"429cee8e-bd15-4c14-800d-8e48377c9ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9" Oct 30 02:27:25.843070 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.843044 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429cee8e-bd15-4c14-800d-8e48377c9ec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d91bc928-pxfr9\" (UID: \"429cee8e-bd15-4c14-800d-8e48377c9ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9" Oct 30 02:27:25.873962 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.873925 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nprxz\" (UniqueName: \"kubernetes.io/projected/429cee8e-bd15-4c14-800d-8e48377c9ec2-kube-api-access-nprxz\") pod \"maas-api-revoke-sub-d91bc928-pxfr9\" (UID: \"429cee8e-bd15-4c14-800d-8e48377c9ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9" Oct 30 02:27:25.876710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:25.876676 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9" Oct 30 02:27:26.031112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:26.012456 5998 patch_prober.go:29] interesting pod/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 02:27:26.031112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:26.012515 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" podUID="2fd2791e-dcdd-40a2-be03-95c5c266d99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 02:27:26.136177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:26.135984 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt"] Oct 30 02:27:26.165776 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:26.165742 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ab7efce-zfnpt"] Oct 30 02:27:26.244108 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:26.241335 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9"] Oct 30 02:27:26.345762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:26.345730 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:26.495588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:26.495547 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9" event={"ID":"429cee8e-bd15-4c14-800d-8e48377c9ec2","Type":"ContainerStarted","Data":"13fbfc708f80967d5272a446c5f654a14a76317bee315033875382584daf2c90"} Oct 30 02:27:26.548932 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:26.548899 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5018b551-1174-4f7c-b99f-b8c48154e68e" path="/var/lib/kubelet/pods/5018b551-1174-4f7c-b99f-b8c48154e68e/volumes" Oct 30 02:27:27.018115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:27.013434 5998 patch_prober.go:29] interesting pod/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 02:27:27.018115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:27.013494 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" podUID="2fd2791e-dcdd-40a2-be03-95c5c266d99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 02:27:27.530110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:27.529416 5998 generic.go:356] "Generic (PLEG): container finished" podID="429cee8e-bd15-4c14-800d-8e48377c9ec2" containerID="8e9dea2da59e462a4892b85e49babd3c694f73154fcaf5fda80322ccafd8bcc4" exitCode=0 Oct 30 02:27:27.530110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:27.529597 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9" event={"ID":"429cee8e-bd15-4c14-800d-8e48377c9ec2","Type":"ContainerDied","Data":"8e9dea2da59e462a4892b85e49babd3c694f73154fcaf5fda80322ccafd8bcc4"} Oct 30 02:27:28.018110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:28.011555 5998 patch_prober.go:29] interesting pod/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 02:27:28.018110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:28.011642 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" podUID="2fd2791e-dcdd-40a2-be03-95c5c266d99c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 02:27:28.083217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:28.083170 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:28.790180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:28.790149 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9" Oct 30 02:27:28.852187 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:28.851637 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/429cee8e-bd15-4c14-800d-8e48377c9ec2-kube-api-access-nprxz\" (UniqueName: \"kubernetes.io/projected/429cee8e-bd15-4c14-800d-8e48377c9ec2-kube-api-access-nprxz\") pod \"429cee8e-bd15-4c14-800d-8e48377c9ec2\" (UID: \"429cee8e-bd15-4c14-800d-8e48377c9ec2\") " Oct 30 02:27:28.852187 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:28.851727 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/429cee8e-bd15-4c14-800d-8e48377c9ec2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429cee8e-bd15-4c14-800d-8e48377c9ec2-maas-api-xks-ca\") pod \"429cee8e-bd15-4c14-800d-8e48377c9ec2\" (UID: \"429cee8e-bd15-4c14-800d-8e48377c9ec2\") " Oct 30 02:27:28.852187 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:28.851836 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/429cee8e-bd15-4c14-800d-8e48377c9ec2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429cee8e-bd15-4c14-800d-8e48377c9ec2-maas-api-openshift-ca\") pod \"429cee8e-bd15-4c14-800d-8e48377c9ec2\" (UID: \"429cee8e-bd15-4c14-800d-8e48377c9ec2\") " Oct 30 02:27:28.854249 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:28.854206 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429cee8e-bd15-4c14-800d-8e48377c9ec2-maas-api-openshift-ca" pod "429cee8e-bd15-4c14-800d-8e48377c9ec2" (UID: "429cee8e-bd15-4c14-800d-8e48377c9ec2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:27:28.855957 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:28.855926 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429cee8e-bd15-4c14-800d-8e48377c9ec2-kube-api-access-nprxz" pod "429cee8e-bd15-4c14-800d-8e48377c9ec2" (UID: "429cee8e-bd15-4c14-800d-8e48377c9ec2"). InnerVolumeSpecName "kube-api-access-nprxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:27:28.856429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:28.856400 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/429cee8e-bd15-4c14-800d-8e48377c9ec2-maas-api-xks-ca" pod "429cee8e-bd15-4c14-800d-8e48377c9ec2" (UID: "429cee8e-bd15-4c14-800d-8e48377c9ec2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:27:28.955534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:28.955493 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/429cee8e-bd15-4c14-800d-8e48377c9ec2-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:28.955534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:28.955526 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nprxz\" (UniqueName: \"kubernetes.io/projected/429cee8e-bd15-4c14-800d-8e48377c9ec2-kube-api-access-nprxz\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:28.955534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:28.955537 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/429cee8e-bd15-4c14-800d-8e48377c9ec2-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:29.096942 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:29.096911 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:27:29.098470 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:29.098446 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:27:29.099522 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:29.099498 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:27:29.558482 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:29.558453 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9" Oct 30 02:27:29.558685 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:29.558484 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9" event={"ID":"429cee8e-bd15-4c14-800d-8e48377c9ec2","Type":"ContainerDied","Data":"13fbfc708f80967d5272a446c5f654a14a76317bee315033875382584daf2c90"} Oct 30 02:27:29.558685 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:29.558533 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13fbfc708f80967d5272a446c5f654a14a76317bee315033875382584daf2c90" Oct 30 02:27:29.805383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:29.805336 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:31.464940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:31.464905 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:32.974173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:32.974137 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:33.655374 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:33.655337 5998 scope.go:122] "RemoveContainer" containerID="d6e94e1a5b2feb5347978c60628d517d083cdff6326b8cef9195a2740af0a2e6" Oct 30 02:27:33.666213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:33.666188 5998 scope.go:122] "RemoveContainer" containerID="c0e3ab1ae466790562d93f3af408abc9f39f3f38de0e1956c81ef9c635d6f441" Oct 30 02:27:34.666541 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:34.666464 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:34.936853 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:34.936655 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x"] Oct 30 02:27:34.938028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:34.937999 5998 cpu_manager.go:414] "containerMap: removing container" podUID="429cee8e-bd15-4c14-800d-8e48377c9ec2" containerName="revoke-keys" Oct 30 02:27:34.938028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:34.938027 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="429cee8e-bd15-4c14-800d-8e48377c9ec2" containerName="revoke-keys" Oct 30 02:27:34.938287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:34.938273 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="429cee8e-bd15-4c14-800d-8e48377c9ec2" containerName="revoke-keys" Oct 30 02:27:34.948380 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:34.948353 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:34.967400 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:34.967367 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x"] Oct 30 02:27:34.969424 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:34.969401 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d97e4c\"/\"openshift-service-ca.crt\"" Oct 30 02:27:34.969614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:34.969420 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d97e4c\"/\"test-model-d97e4c-kserve-self-signed-certs\"" Oct 30 02:27:34.972123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:34.972100 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d97e4c\"/\"kube-root-ca.crt\"" Oct 30 02:27:34.972226 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:34.972139 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d97e4c\"/\"default-dockercfg-z9kv8\"" Oct 30 02:27:34.977962 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:34.977923 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-model-cache\") pod \"test-model-d97e4c-kserve-66465b5c75-zgz5x\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:34.978112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:34.977966 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcq4l\" (UniqueName: \"kubernetes.io/projected/14b0f602-68d2-4494-a92f-c6df12bcdaad-kube-api-access-lcq4l\") pod \"test-model-d97e4c-kserve-66465b5c75-zgz5x\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:34.978906 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:34.978230 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-tmp-dir\") pod \"test-model-d97e4c-kserve-66465b5c75-zgz5x\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:34.978906 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:34.978387 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-home\") pod \"test-model-d97e4c-kserve-66465b5c75-zgz5x\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:34.978906 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:34.978468 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-dshm\") pod \"test-model-d97e4c-kserve-66465b5c75-zgz5x\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:34.978906 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:34.978496 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14b0f602-68d2-4494-a92f-c6df12bcdaad-tls-certs\") pod \"test-model-d97e4c-kserve-66465b5c75-zgz5x\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:34.989023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:34.988986 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:35.079867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:35.079824 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-dshm\") pod \"test-model-d97e4c-kserve-66465b5c75-zgz5x\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:35.080076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:35.079873 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14b0f602-68d2-4494-a92f-c6df12bcdaad-tls-certs\") pod \"test-model-d97e4c-kserve-66465b5c75-zgz5x\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:35.080076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:35.080045 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-model-cache\") pod \"test-model-d97e4c-kserve-66465b5c75-zgz5x\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:35.080076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:35.080068 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcq4l\" (UniqueName: \"kubernetes.io/projected/14b0f602-68d2-4494-a92f-c6df12bcdaad-kube-api-access-lcq4l\") pod \"test-model-d97e4c-kserve-66465b5c75-zgz5x\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:35.080289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:35.080214 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-tmp-dir\") pod \"test-model-d97e4c-kserve-66465b5c75-zgz5x\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:35.080350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:35.080335 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-home\") pod \"test-model-d97e4c-kserve-66465b5c75-zgz5x\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:35.080467 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:35.080441 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-model-cache\") pod \"test-model-d97e4c-kserve-66465b5c75-zgz5x\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:35.080792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:35.080764 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-home\") pod \"test-model-d97e4c-kserve-66465b5c75-zgz5x\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:35.080919 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:35.080782 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-tmp-dir\") pod \"test-model-d97e4c-kserve-66465b5c75-zgz5x\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:35.082573 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:35.082550 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-dshm\") pod \"test-model-d97e4c-kserve-66465b5c75-zgz5x\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:35.082746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:35.082728 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14b0f602-68d2-4494-a92f-c6df12bcdaad-tls-certs\") pod \"test-model-d97e4c-kserve-66465b5c75-zgz5x\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:35.103850 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:35.103822 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcq4l\" (UniqueName: \"kubernetes.io/projected/14b0f602-68d2-4494-a92f-c6df12bcdaad-kube-api-access-lcq4l\") pod \"test-model-d97e4c-kserve-66465b5c75-zgz5x\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:35.274175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:35.273426 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:35.691867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:35.691839 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x"] Oct 30 02:27:35.694302 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:27:35.694267 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14b0f602_68d2_4494_a92f_c6df12bcdaad.slice/crio-1fcd21bf4f11e21bb64f0391abdb5fd3038c986c8f3f2dca68bf51bd5ab87e29 WatchSource:0}: Error finding container 1fcd21bf4f11e21bb64f0391abdb5fd3038c986c8f3f2dca68bf51bd5ab87e29: Status 404 returned error can't find the container with id 1fcd21bf4f11e21bb64f0391abdb5fd3038c986c8f3f2dca68bf51bd5ab87e29 Oct 30 02:27:36.357573 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:36.357518 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:36.637557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:36.637454 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" event={"ID":"14b0f602-68d2-4494-a92f-c6df12bcdaad","Type":"ContainerStarted","Data":"d24c607f64f2863560cb023e70d6a4e2bef540daad0b86a4fb6612121a66d11e"} Oct 30 02:27:36.637557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:36.637511 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" event={"ID":"14b0f602-68d2-4494-a92f-c6df12bcdaad","Type":"ContainerStarted","Data":"1fcd21bf4f11e21bb64f0391abdb5fd3038c986c8f3f2dca68bf51bd5ab87e29"} Oct 30 02:27:36.682372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:36.682309 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" podStartSLOduration=2.682289505 podStartE2EDuration="2.682289505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:27:36.677735899 +0000 UTC m=+3486.256526981" watchObservedRunningTime="2026-10-30 02:27:36.682289505 +0000 UTC m=+3486.261080591" Oct 30 02:27:37.261418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:37.261382 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:37.274009 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:37.273971 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:37.403115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:37.399783 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:37.675221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:37.675183 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:38.091963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.091851 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:38.342166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.284167 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn"] Oct 30 02:27:38.448030 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.410542 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn"] Oct 30 02:27:38.448030 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.410682 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn" Oct 30 02:27:38.448030 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.439188 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:27:38.448030 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.439896 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:27:38.452662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.452634 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:27:38.453083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.453063 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:27:38.467705 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.467665 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:38.586995 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.470193 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae632736-734a-47bd-9987-dc725dbf3b11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-615952d9-vlngn\" (UID: \"ae632736-734a-47bd-9987-dc725dbf3b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn" Oct 30 02:27:38.587222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.587201 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvkxc\" (UniqueName: \"kubernetes.io/projected/ae632736-734a-47bd-9987-dc725dbf3b11-kube-api-access-cvkxc\") pod \"maas-api-revoke-sub-615952d9-vlngn\" (UID: \"ae632736-734a-47bd-9987-dc725dbf3b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn" Oct 30 02:27:38.587275 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.587249 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae632736-734a-47bd-9987-dc725dbf3b11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-615952d9-vlngn\" (UID: \"ae632736-734a-47bd-9987-dc725dbf3b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn" Oct 30 02:27:38.638285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.638187 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb"] Oct 30 02:27:38.644148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.644119 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.663012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.662978 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d207e8-openshift-default-dockercfg-q4chv\"" Oct 30 02:27:38.665256 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.665215 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb"] Oct 30 02:27:38.689148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.688536 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-istio-data\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.689819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.689398 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-istio-envoy\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.690056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.690028 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-credential-socket\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.690205 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.690154 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thbm4\" (UniqueName: \"kubernetes.io/projected/49169ae5-7185-44fc-9c12-de093c4a1510-kube-api-access-thbm4\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.690273 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.690234 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae632736-734a-47bd-9987-dc725dbf3b11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-615952d9-vlngn\" (UID: \"ae632736-734a-47bd-9987-dc725dbf3b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn" Oct 30 02:27:38.690333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.690271 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49169ae5-7185-44fc-9c12-de093c4a1510-istiod-ca-cert\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.690459 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.690443 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-workload-socket\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.690528 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.690476 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-workload-certs\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.690528 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.690516 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49169ae5-7185-44fc-9c12-de093c4a1510-istio-token\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.690636 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.690556 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49169ae5-7185-44fc-9c12-de093c4a1510-istio-podinfo\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.691014 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.690991 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvkxc\" (UniqueName: \"kubernetes.io/projected/ae632736-734a-47bd-9987-dc725dbf3b11-kube-api-access-cvkxc\") pod \"maas-api-revoke-sub-615952d9-vlngn\" (UID: \"ae632736-734a-47bd-9987-dc725dbf3b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn" Oct 30 02:27:38.691234 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.691204 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae632736-734a-47bd-9987-dc725dbf3b11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-615952d9-vlngn\" (UID: \"ae632736-734a-47bd-9987-dc725dbf3b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn" Oct 30 02:27:38.692527 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.692476 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae632736-734a-47bd-9987-dc725dbf3b11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-615952d9-vlngn\" (UID: \"ae632736-734a-47bd-9987-dc725dbf3b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn" Oct 30 02:27:38.694480 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.694457 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae632736-734a-47bd-9987-dc725dbf3b11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-615952d9-vlngn\" (UID: \"ae632736-734a-47bd-9987-dc725dbf3b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn" Oct 30 02:27:38.803203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.798682 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:38.830904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.830863 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-istio-data\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.831223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.831195 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-istio-envoy\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.831409 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.831392 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-credential-socket\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.831606 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.831588 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thbm4\" (UniqueName: \"kubernetes.io/projected/49169ae5-7185-44fc-9c12-de093c4a1510-kube-api-access-thbm4\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.837109 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.831747 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49169ae5-7185-44fc-9c12-de093c4a1510-istiod-ca-cert\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.837458 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.837431 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-workload-socket\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.837612 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.837596 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-workload-certs\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.837743 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.837727 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49169ae5-7185-44fc-9c12-de093c4a1510-istio-token\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.837869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.837851 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49169ae5-7185-44fc-9c12-de093c4a1510-istio-podinfo\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.900162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.874878 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvkxc\" (UniqueName: \"kubernetes.io/projected/ae632736-734a-47bd-9987-dc725dbf3b11-kube-api-access-cvkxc\") pod \"maas-api-revoke-sub-615952d9-vlngn\" (UID: \"ae632736-734a-47bd-9987-dc725dbf3b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn" Oct 30 02:27:38.900414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.875463 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-istio-data\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.900522 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.876382 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49169ae5-7185-44fc-9c12-de093c4a1510-istiod-ca-cert\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.900614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.876617 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-credential-socket\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.900706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.877187 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-workload-certs\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.900791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.877418 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-workload-socket\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.900886 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.885207 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49169ae5-7185-44fc-9c12-de093c4a1510-istio-podinfo\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.919130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.886247 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-istio-envoy\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.942111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.939452 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49169ae5-7185-44fc-9c12-de093c4a1510-istio-token\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.942111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.940185 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thbm4\" (UniqueName: \"kubernetes.io/projected/49169ae5-7185-44fc-9c12-de093c4a1510-kube-api-access-thbm4\") pod \"e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:38.964115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.943658 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:38.967120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:38.967076 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:39.168028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:39.167485 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:39.168028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:39.167796 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn" Oct 30 02:27:39.575467 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:39.575399 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn"] Oct 30 02:27:39.584659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:39.584612 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb"] Oct 30 02:27:39.586013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:39.585984 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:27:39.586144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:39.586046 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:27:39.586144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:39.586111 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:27:39.675109 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:39.672752 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" event={"ID":"49169ae5-7185-44fc-9c12-de093c4a1510","Type":"ContainerStarted","Data":"5845ce36782e34f2bb7d682b1b61bde6f891f4ed1388effb1b74f1ca4588bdf3"} Oct 30 02:27:39.675109 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:39.674828 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn" event={"ID":"ae632736-734a-47bd-9987-dc725dbf3b11","Type":"ContainerStarted","Data":"84bdbd4e88e07b8a50d23cc277de943694080a397c86b12effac930c054c8b43"} Oct 30 02:27:40.479328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:40.479290 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:40.685150 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:40.685081 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn" event={"ID":"ae632736-734a-47bd-9987-dc725dbf3b11","Type":"ContainerStarted","Data":"90a67e29b23d6d65049494bbcc8b7c67dec1df6e5be8193599a4a337df17cbe4"} Oct 30 02:27:40.687243 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:40.687213 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" event={"ID":"49169ae5-7185-44fc-9c12-de093c4a1510","Type":"ContainerStarted","Data":"7089b6c23ec1cc5a857889536a8e186ae6a5c177bde932a77d9f1b28205e1e0e"} Oct 30 02:27:40.729636 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:40.729548 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn" podStartSLOduration=2.729532234 podStartE2EDuration="2.729532234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:27:40.725406033 +0000 UTC m=+3490.304197113" watchObservedRunningTime="2026-10-30 02:27:40.729532234 +0000 UTC m=+3490.308323314" Oct 30 02:27:40.780438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:40.780379 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" podStartSLOduration=2.78036399 podStartE2EDuration="2.78036399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:27:40.7756593 +0000 UTC m=+3490.354450383" watchObservedRunningTime="2026-10-30 02:27:40.78036399 +0000 UTC m=+3490.359155071" Oct 30 02:27:40.968748 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:40.968707 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:40.977431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:40.977401 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:41.750027 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:41.749942 5998 generic.go:356] "Generic (PLEG): container finished" podID="ae632736-734a-47bd-9987-dc725dbf3b11" containerID="90a67e29b23d6d65049494bbcc8b7c67dec1df6e5be8193599a4a337df17cbe4" exitCode=0 Oct 30 02:27:41.751522 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:41.751489 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn" event={"ID":"ae632736-734a-47bd-9987-dc725dbf3b11","Type":"ContainerDied","Data":"90a67e29b23d6d65049494bbcc8b7c67dec1df6e5be8193599a4a337df17cbe4"} Oct 30 02:27:41.751694 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:41.751682 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:41.768352 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:41.768282 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:27:42.472521 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:42.472487 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:42.934000 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:42.933976 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn" Oct 30 02:27:43.020383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:43.020298 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae632736-734a-47bd-9987-dc725dbf3b11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae632736-734a-47bd-9987-dc725dbf3b11-maas-api-openshift-ca\") pod \"ae632736-734a-47bd-9987-dc725dbf3b11\" (UID: \"ae632736-734a-47bd-9987-dc725dbf3b11\") " Oct 30 02:27:43.020596 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:43.020571 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae632736-734a-47bd-9987-dc725dbf3b11-kube-api-access-cvkxc\" (UniqueName: \"kubernetes.io/projected/ae632736-734a-47bd-9987-dc725dbf3b11-kube-api-access-cvkxc\") pod \"ae632736-734a-47bd-9987-dc725dbf3b11\" (UID: \"ae632736-734a-47bd-9987-dc725dbf3b11\") " Oct 30 02:27:43.020782 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:43.020755 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae632736-734a-47bd-9987-dc725dbf3b11-maas-api-openshift-ca" pod "ae632736-734a-47bd-9987-dc725dbf3b11" (UID: "ae632736-734a-47bd-9987-dc725dbf3b11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:27:43.021271 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:43.021244 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae632736-734a-47bd-9987-dc725dbf3b11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae632736-734a-47bd-9987-dc725dbf3b11-maas-api-xks-ca\") pod \"ae632736-734a-47bd-9987-dc725dbf3b11\" (UID: \"ae632736-734a-47bd-9987-dc725dbf3b11\") " Oct 30 02:27:43.021962 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:43.021936 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae632736-734a-47bd-9987-dc725dbf3b11-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:43.023239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:43.023206 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae632736-734a-47bd-9987-dc725dbf3b11-kube-api-access-cvkxc" pod "ae632736-734a-47bd-9987-dc725dbf3b11" (UID: "ae632736-734a-47bd-9987-dc725dbf3b11"). InnerVolumeSpecName "kube-api-access-cvkxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:27:43.023814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:43.023785 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae632736-734a-47bd-9987-dc725dbf3b11-maas-api-xks-ca" pod "ae632736-734a-47bd-9987-dc725dbf3b11" (UID: "ae632736-734a-47bd-9987-dc725dbf3b11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:27:43.123757 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:43.123720 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvkxc\" (UniqueName: \"kubernetes.io/projected/ae632736-734a-47bd-9987-dc725dbf3b11-kube-api-access-cvkxc\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:43.123757 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:43.123752 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae632736-734a-47bd-9987-dc725dbf3b11-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:43.167906 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:43.167874 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:43.323328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:43.323245 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:43.766621 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:43.766588 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn" Oct 30 02:27:43.766621 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:43.766616 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn" event={"ID":"ae632736-734a-47bd-9987-dc725dbf3b11","Type":"ContainerDied","Data":"84bdbd4e88e07b8a50d23cc277de943694080a397c86b12effac930c054c8b43"} Oct 30 02:27:43.766899 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:43.766681 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84bdbd4e88e07b8a50d23cc277de943694080a397c86b12effac930c054c8b43" Oct 30 02:27:44.389245 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:44.389210 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:44.890901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:44.890866 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:45.036879 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:45.036832 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:45.290305 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:45.283918 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:45.526964 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:45.520251 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:45.795604 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:45.795572 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:27:46.143198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:46.143107 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:47.779274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:47.779238 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:49.316307 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.316260 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6"] Oct 30 02:27:49.317602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.317577 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ae632736-734a-47bd-9987-dc725dbf3b11" containerName="revoke-keys" Oct 30 02:27:49.317722 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.317604 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae632736-734a-47bd-9987-dc725dbf3b11" containerName="revoke-keys" Oct 30 02:27:49.317841 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.317829 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae632736-734a-47bd-9987-dc725dbf3b11" containerName="revoke-keys" Oct 30 02:27:49.322147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.322125 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6" Oct 30 02:27:49.335385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.335359 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:27:49.341039 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.341007 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6"] Oct 30 02:27:49.345934 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.345910 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:27:49.346073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.345998 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:27:49.346073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.346012 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:27:49.376645 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.376602 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f00a4123-txnq6\" (UID: \"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6" Oct 30 02:27:49.377061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.377033 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4x7c\" (UniqueName: \"kubernetes.io/projected/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-kube-api-access-v4x7c\") pod \"maas-api-revoke-sub-f00a4123-txnq6\" (UID: \"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6" Oct 30 02:27:49.377246 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.377228 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f00a4123-txnq6\" (UID: \"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6" Oct 30 02:27:49.386334 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.386293 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:49.479797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.479758 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4x7c\" (UniqueName: \"kubernetes.io/projected/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-kube-api-access-v4x7c\") pod \"maas-api-revoke-sub-f00a4123-txnq6\" (UID: \"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6" Oct 30 02:27:49.480019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.479805 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f00a4123-txnq6\" (UID: \"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6" Oct 30 02:27:49.480105 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.480032 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f00a4123-txnq6\" (UID: \"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6" Oct 30 02:27:49.481497 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.481464 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f00a4123-txnq6\" (UID: \"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6" Oct 30 02:27:49.483102 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.483054 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f00a4123-txnq6\" (UID: \"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6" Oct 30 02:27:49.501854 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.501824 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4x7c\" (UniqueName: \"kubernetes.io/projected/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-kube-api-access-v4x7c\") pod \"maas-api-revoke-sub-f00a4123-txnq6\" (UID: \"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6" Oct 30 02:27:49.639588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.639492 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6" Oct 30 02:27:49.981971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:49.981939 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6"] Oct 30 02:27:49.983959 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:27:49.983930 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1df1953_76a5_4cb5_a2d1_0fc791c6a27c.slice/crio-da6091ae38c682d02a4e792af7c9f7183b1a2f0c4462dc0bf9e077e1c3d9227c WatchSource:0}: Error finding container da6091ae38c682d02a4e792af7c9f7183b1a2f0c4462dc0bf9e077e1c3d9227c: Status 404 returned error can't find the container with id da6091ae38c682d02a4e792af7c9f7183b1a2f0c4462dc0bf9e077e1c3d9227c Oct 30 02:27:50.828201 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:50.828155 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6" event={"ID":"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c","Type":"ContainerStarted","Data":"db04f6788fc8f5d2b78247639256e064fba851a390ddf7257165263252464949"} Oct 30 02:27:50.828656 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:50.828229 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6" event={"ID":"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c","Type":"ContainerStarted","Data":"da6091ae38c682d02a4e792af7c9f7183b1a2f0c4462dc0bf9e077e1c3d9227c"} Oct 30 02:27:50.872478 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:50.872424 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6" podStartSLOduration=1.872406929 podStartE2EDuration="1.872406929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:27:50.868296786 +0000 UTC m=+3500.447087866" watchObservedRunningTime="2026-10-30 02:27:50.872406929 +0000 UTC m=+3500.451198009" Oct 30 02:27:50.992851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:50.992821 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:51.839800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:51.839763 5998 generic.go:356] "Generic (PLEG): container finished" podID="a1df1953-76a5-4cb5-a2d1-0fc791c6a27c" containerID="db04f6788fc8f5d2b78247639256e064fba851a390ddf7257165263252464949" exitCode=0 Oct 30 02:27:51.840347 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:51.839845 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6" event={"ID":"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c","Type":"ContainerDied","Data":"db04f6788fc8f5d2b78247639256e064fba851a390ddf7257165263252464949"} Oct 30 02:27:52.580791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:52.580758 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:53.040802 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:53.040768 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6" Oct 30 02:27:53.158701 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:53.158609 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-maas-api-openshift-ca\") pod \"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c\" (UID: \"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c\") " Oct 30 02:27:53.158894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:53.158739 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-maas-api-xks-ca\") pod \"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c\" (UID: \"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c\") " Oct 30 02:27:53.158894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:53.158803 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-kube-api-access-v4x7c\" (UniqueName: \"kubernetes.io/projected/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-kube-api-access-v4x7c\") pod \"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c\" (UID: \"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c\") " Oct 30 02:27:53.159007 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:53.158980 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-maas-api-openshift-ca" pod "a1df1953-76a5-4cb5-a2d1-0fc791c6a27c" (UID: "a1df1953-76a5-4cb5-a2d1-0fc791c6a27c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:27:53.159512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:53.159486 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:53.160862 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:53.160831 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-kube-api-access-v4x7c" pod "a1df1953-76a5-4cb5-a2d1-0fc791c6a27c" (UID: "a1df1953-76a5-4cb5-a2d1-0fc791c6a27c"). InnerVolumeSpecName "kube-api-access-v4x7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:27:53.160982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:53.160885 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-maas-api-xks-ca" pod "a1df1953-76a5-4cb5-a2d1-0fc791c6a27c" (UID: "a1df1953-76a5-4cb5-a2d1-0fc791c6a27c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:27:53.260842 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:53.260811 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:53.260842 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:53.260840 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4x7c\" (UniqueName: \"kubernetes.io/projected/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c-kube-api-access-v4x7c\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:27:53.858186 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:53.858137 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6" event={"ID":"a1df1953-76a5-4cb5-a2d1-0fc791c6a27c","Type":"ContainerDied","Data":"da6091ae38c682d02a4e792af7c9f7183b1a2f0c4462dc0bf9e077e1c3d9227c"} Oct 30 02:27:53.858347 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:53.858213 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da6091ae38c682d02a4e792af7c9f7183b1a2f0c4462dc0bf9e077e1c3d9227c" Oct 30 02:27:53.858347 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:53.858153 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6" Oct 30 02:27:54.094178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:54.094147 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:55.223307 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:27:55.223262 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:27:55.724933 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:55.724901 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:57.397343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:57.397309 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:58.986317 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:58.986278 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:27:59.086172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:59.082476 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q"] Oct 30 02:27:59.090361 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:27:59.090325 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b4faac4-jzn4q"] Oct 30 02:28:00.527618 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:00.527585 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:00.543766 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:00.543734 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6142a278-891e-4516-820c-4e6c3b29fac0" path="/var/lib/kubelet/pods/6142a278-891e-4516-820c-4e6c3b29fac0/volumes" Oct 30 02:28:00.639688 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:00.639651 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:00.750658 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:00.750624 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:00.962552 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:00.962511 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:01.074624 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:01.074587 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:01.360267 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:01.360182 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:01.556172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:01.556126 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:02.356581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:02.356526 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:02.491001 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:02.490963 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:04.088977 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:04.088934 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:05.778785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:05.778734 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:07.274847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:07.274808 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:08.354879 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:08.354836 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:08.480408 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:08.480370 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:08.855007 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:08.854974 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:08.989178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:08.989141 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:09.482114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.481072 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j"] Oct 30 02:28:09.482643 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.482138 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a1df1953-76a5-4cb5-a2d1-0fc791c6a27c" containerName="revoke-keys" Oct 30 02:28:09.482643 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.482159 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1df1953-76a5-4cb5-a2d1-0fc791c6a27c" containerName="revoke-keys" Oct 30 02:28:09.482643 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.482315 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1df1953-76a5-4cb5-a2d1-0fc791c6a27c" containerName="revoke-keys" Oct 30 02:28:09.513349 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.506857 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.545114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.540544 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-52d131-openshift-default-dockercfg-8vc6p\"" Oct 30 02:28:09.564864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.564828 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j"] Oct 30 02:28:09.578193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.577615 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:09.717688 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.642537 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-token\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.791336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.791244 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-credential-socket\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.791704 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.791675 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-data\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.791872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.791851 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-podinfo\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.792070 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.792053 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-workload-socket\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.792247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.792228 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-envoy\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.792447 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.792428 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istiod-ca-cert\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.792587 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.792569 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j5gs\" (UniqueName: \"kubernetes.io/projected/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-kube-api-access-9j5gs\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.792777 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.792760 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-workload-certs\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.793083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.749981 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:09.894284 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.894247 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-token\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.894511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.894292 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-credential-socket\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.894511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.894391 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-data\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.894511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.894437 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-podinfo\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.894704 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.894590 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-workload-socket\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.894704 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.894628 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-envoy\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.894826 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.894699 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istiod-ca-cert\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.894826 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.894735 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j5gs\" (UniqueName: \"kubernetes.io/projected/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-kube-api-access-9j5gs\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.894945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.894866 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-data\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.894991 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.894936 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-workload-certs\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.895419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.895374 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-workload-socket\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.895526 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.895441 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-credential-socket\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.895592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.895575 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-workload-certs\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.895888 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.895867 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istiod-ca-cert\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.897322 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.897295 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-podinfo\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.897745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.897722 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-envoy\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.917378 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.917342 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j5gs\" (UniqueName: \"kubernetes.io/projected/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-kube-api-access-9j5gs\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.917789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.917767 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-token\") pod \"e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:09.998178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:09.998140 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:10.042701 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:10.042619 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq"] Oct 30 02:28:10.042968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:10.042951 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a94b656-xbqqq"] Oct 30 02:28:10.339045 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:10.339011 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j"] Oct 30 02:28:10.340684 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:28:10.340650 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea4d2cb_cdaa_4ce4_abee_c2866af53c2c.slice/crio-b909365d4a3c22529b9779a609a31c68dc12554c0e0ccad214c702cf1d1332fc WatchSource:0}: Error finding container b909365d4a3c22529b9779a609a31c68dc12554c0e0ccad214c702cf1d1332fc: Status 404 returned error can't find the container with id b909365d4a3c22529b9779a609a31c68dc12554c0e0ccad214c702cf1d1332fc Oct 30 02:28:10.344926 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:10.344884 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:28:10.345037 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:10.344978 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:28:10.345037 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:10.345025 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:28:10.551742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:10.551709 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a66c194f-7609-49ca-9fcc-81285e0b60c8" path="/var/lib/kubelet/pods/a66c194f-7609-49ca-9fcc-81285e0b60c8/volumes" Oct 30 02:28:10.746454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:10.746393 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:11.123856 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:11.123749 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" event={"ID":"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c","Type":"ContainerStarted","Data":"c1eaa521288148421bc1583339bf04077523312cebfc7375f7db69d923bdcfa5"} Oct 30 02:28:11.123856 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:11.123815 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" event={"ID":"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c","Type":"ContainerStarted","Data":"b909365d4a3c22529b9779a609a31c68dc12554c0e0ccad214c702cf1d1332fc"} Oct 30 02:28:11.173083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:11.173015 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" podStartSLOduration=2.172994413 podStartE2EDuration="2.172994413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:28:11.165808528 +0000 UTC m=+3520.744599610" watchObservedRunningTime="2026-10-30 02:28:11.172994413 +0000 UTC m=+3520.751785495" Oct 30 02:28:11.999240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:11.999147 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:12.006855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:12.006823 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:12.136315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:12.136283 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:12.137400 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:12.137376 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:28:12.407833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:12.407747 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:13.992815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:13.992775 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:15.502218 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:15.502180 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:17.085213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:17.085181 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:17.204962 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:17.204921 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:17.348346 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:17.348243 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:18.790405 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:18.790364 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:18.910755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:18.910713 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:20.145114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:20.120055 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:28:20.509118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:20.509060 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:21.755792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:21.754186 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:21.893345 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:21.893299 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:22.069073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:22.068980 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:23.084833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:23.084797 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:23.506407 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:23.506363 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:23.619626 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:23.619588 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:24.863816 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:24.863780 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:25.036074 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:25.036031 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6"] Oct 30 02:28:25.045503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:25.045463 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4e747a6-zcjd6"] Oct 30 02:28:25.240698 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:28:25.240660 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:28:26.463330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:26.463298 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:26.566154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:26.566115 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="032e9f87-821b-4408-9407-0ed3ce36dbfb" path="/var/lib/kubelet/pods/032e9f87-821b-4408-9407-0ed3ce36dbfb/volumes" Oct 30 02:28:28.096468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:28.096404 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:29.610083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:29.610043 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:31.180570 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:31.180532 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:32.760158 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:32.760116 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:33.704312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:33.704280 5998 scope.go:122] "RemoveContainer" containerID="9bb9f41b1fab6074d3fea1974e5ca8a6df5105db2613d722de6394440bc4fbfa" Oct 30 02:28:34.355237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:34.355202 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:36.089768 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:36.089704 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:37.729440 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:37.729400 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:37.873255 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:37.873216 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:38.028051 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:38.027970 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:39.290789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:39.290756 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:39.502208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:39.502171 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:41.194251 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:41.194216 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:42.863761 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:42.863728 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:44.514246 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:44.514213 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:46.160804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:46.160771 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:47.724937 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:47.724898 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:49.267711 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:49.267675 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:51.017079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:51.017041 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:52.686270 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:52.686236 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:54.294269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:54.294233 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:55.220738 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:28:55.220690 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:28:55.891661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:55.891628 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:57.422765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:57.422728 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:28:59.104693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:28:59.104654 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:01.133593 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:01.133532 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:02.714777 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:02.714720 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:03.688422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:03.683656 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:29:04.077054 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:04.076964 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:04.433544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:04.433504 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:05.912048 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:05.912012 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz"] Oct 30 02:29:05.916460 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:05.916442 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:05.943472 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:05.943446 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2943ff\"/\"test-model-2943ff-kserve-self-signed-certs\"" Oct 30 02:29:05.943472 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:05.943483 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2943ff\"/\"default-dockercfg-zddjs\"" Oct 30 02:29:05.943988 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:05.943968 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2943ff\"/\"openshift-service-ca.crt\"" Oct 30 02:29:05.944054 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:05.943992 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2943ff\"/\"kube-root-ca.crt\"" Oct 30 02:29:05.955256 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:05.955229 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz"] Oct 30 02:29:06.029958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.029914 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86ng9\" (UniqueName: \"kubernetes.io/projected/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-kube-api-access-86ng9\") pod \"test-model-2943ff-kserve-9c9949c6c-kwfvz\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.030169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.029972 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-home\") pod \"test-model-2943ff-kserve-9c9949c6c-kwfvz\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.030169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.030040 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-tmp-dir\") pod \"test-model-2943ff-kserve-9c9949c6c-kwfvz\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.030264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.030196 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-model-cache\") pod \"test-model-2943ff-kserve-9c9949c6c-kwfvz\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.030370 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.030353 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-tls-certs\") pod \"test-model-2943ff-kserve-9c9949c6c-kwfvz\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.030479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.030462 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-dshm\") pod \"test-model-2943ff-kserve-9c9949c6c-kwfvz\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.090254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.090218 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:06.132022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.131985 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-dshm\") pod \"test-model-2943ff-kserve-9c9949c6c-kwfvz\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.132234 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.132216 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86ng9\" (UniqueName: \"kubernetes.io/projected/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-kube-api-access-86ng9\") pod \"test-model-2943ff-kserve-9c9949c6c-kwfvz\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.132295 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.132245 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-home\") pod \"test-model-2943ff-kserve-9c9949c6c-kwfvz\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.132295 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.132263 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-tmp-dir\") pod \"test-model-2943ff-kserve-9c9949c6c-kwfvz\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.132414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.132321 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-model-cache\") pod \"test-model-2943ff-kserve-9c9949c6c-kwfvz\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.132477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.132451 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-tls-certs\") pod \"test-model-2943ff-kserve-9c9949c6c-kwfvz\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.132714 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.132684 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-home\") pod \"test-model-2943ff-kserve-9c9949c6c-kwfvz\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.132940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.132910 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-tmp-dir\") pod \"test-model-2943ff-kserve-9c9949c6c-kwfvz\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.133063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.132951 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-model-cache\") pod \"test-model-2943ff-kserve-9c9949c6c-kwfvz\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.134946 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.134922 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-dshm\") pod \"test-model-2943ff-kserve-9c9949c6c-kwfvz\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.135356 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.135333 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-tls-certs\") pod \"test-model-2943ff-kserve-9c9949c6c-kwfvz\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.162566 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.162496 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86ng9\" (UniqueName: \"kubernetes.io/projected/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-kube-api-access-86ng9\") pod \"test-model-2943ff-kserve-9c9949c6c-kwfvz\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.231152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.230681 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:06.558653 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.558500 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:06.787265 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.787068 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz"] Oct 30 02:29:06.978519 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.978477 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" event={"ID":"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02","Type":"ContainerStarted","Data":"d77e875a5dc67dc475632a08d74e48df98b95ef8526ea8e482140526645884a4"} Oct 30 02:29:06.978918 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:06.978533 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" event={"ID":"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02","Type":"ContainerStarted","Data":"313959391aea5179164453a843c78b62584294d5e7d2d661530af9c7aa490e9c"} Oct 30 02:29:07.037175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:07.023254 5998 patch_prober.go:29] interesting pod/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:29:07.037175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:07.023321 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" podUID="025cc8ef-572d-4f6a-851c-2e59fb797189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:29:07.041919 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:07.041856 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" podStartSLOduration=2.041836691 podStartE2EDuration="2.041836691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:29:07.017544163 +0000 UTC m=+3576.596335246" watchObservedRunningTime="2026-10-30 02:29:07.041836691 +0000 UTC m=+3576.620627773" Oct 30 02:29:07.237807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:07.237743 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:08.191332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:08.191288 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:09.914619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:09.914581 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:11.486855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:11.486774 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:12.070428 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:12.070374 5998 generic.go:356] "Generic (PLEG): container finished" podID="025cc8ef-572d-4f6a-851c-2e59fb797189" containerID="1cc92474d04402c331a74bccbbd597eefb521f2411ec393cecf2a3348146d7d5" exitCode=0 Oct 30 02:29:12.070612 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:12.070465 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" event={"ID":"025cc8ef-572d-4f6a-851c-2e59fb797189","Type":"ContainerDied","Data":"1cc92474d04402c331a74bccbbd597eefb521f2411ec393cecf2a3348146d7d5"} Oct 30 02:29:12.071001 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:12.070979 5998 scope.go:122] "RemoveContainer" containerID="1cc92474d04402c331a74bccbbd597eefb521f2411ec393cecf2a3348146d7d5" Oct 30 02:29:12.073293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:12.073266 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:29:12.073392 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:12.073337 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:29:12.073392 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:12.073369 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:29:13.030241 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:13.030181 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:13.124113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:13.120301 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" event={"ID":"025cc8ef-572d-4f6a-851c-2e59fb797189","Type":"ContainerStarted","Data":"2e037cefb4108dd31ce72b75e2897b437490810ae7f5efb2c17123773e729e18"} Oct 30 02:29:13.274583 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:13.274546 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:29:13.274760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:13.274659 5998 patch_prober.go:29] interesting pod/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 02:29:13.274760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:13.274712 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" podUID="025cc8ef-572d-4f6a-851c-2e59fb797189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 02:29:14.274807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:14.274756 5998 patch_prober.go:29] interesting pod/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 02:29:14.275317 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:14.274820 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" podUID="025cc8ef-572d-4f6a-851c-2e59fb797189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 02:29:14.705116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:14.704602 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:15.274458 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:15.274417 5998 patch_prober.go:29] interesting pod/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 02:29:15.274682 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:15.274479 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" podUID="025cc8ef-572d-4f6a-851c-2e59fb797189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 02:29:16.266872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:16.248715 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:16.286326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:16.285838 5998 patch_prober.go:29] interesting pod/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 02:29:16.286326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:16.285904 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" podUID="025cc8ef-572d-4f6a-851c-2e59fb797189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 02:29:16.514120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:16.511034 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:16.526164 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:16.519979 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:17.184838 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:17.184809 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:29:17.275057 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:17.275018 5998 patch_prober.go:29] interesting pod/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 02:29:17.275640 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:17.275080 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" podUID="025cc8ef-572d-4f6a-851c-2e59fb797189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 02:29:18.182675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:18.182640 5998 generic.go:356] "Generic (PLEG): container finished" podID="49169ae5-7185-44fc-9c12-de093c4a1510" containerID="7089b6c23ec1cc5a857889536a8e186ae6a5c177bde932a77d9f1b28205e1e0e" exitCode=0 Oct 30 02:29:18.182907 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:18.182714 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" event={"ID":"49169ae5-7185-44fc-9c12-de093c4a1510","Type":"ContainerDied","Data":"7089b6c23ec1cc5a857889536a8e186ae6a5c177bde932a77d9f1b28205e1e0e"} Oct 30 02:29:18.183527 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:18.183500 5998 scope.go:122] "RemoveContainer" containerID="7089b6c23ec1cc5a857889536a8e186ae6a5c177bde932a77d9f1b28205e1e0e" Oct 30 02:29:18.185823 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:18.185789 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:29:18.185926 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:18.185877 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:29:18.185973 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:18.185923 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:29:18.274970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:18.274927 5998 patch_prober.go:29] interesting pod/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 02:29:18.275175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:18.274983 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" podUID="025cc8ef-572d-4f6a-851c-2e59fb797189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 02:29:18.362541 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:18.362512 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:18.476179 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:18.476078 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:18.606354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:18.603941 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:19.195199 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:19.195137 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" event={"ID":"49169ae5-7185-44fc-9c12-de093c4a1510","Type":"ContainerStarted","Data":"3f194d72314a0a862ae405d5bc92780190597d7b448d03584e88874a95538972"} Oct 30 02:29:19.290116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:19.286035 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:29:19.290116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:19.286900 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:29:19.290116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:19.287828 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:29:19.967978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:19.967947 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:29:19.968294 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:19.968267 5998 patch_prober.go:29] interesting pod/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Oct 30 02:29:19.968356 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:19.968322 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" podUID="49169ae5-7185-44fc-9c12-de093c4a1510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Oct 30 02:29:20.087176 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:20.087142 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:20.428120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:20.426539 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:20.968851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:20.968813 5998 patch_prober.go:29] interesting pod/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Oct 30 02:29:20.969113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:20.968885 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" podUID="49169ae5-7185-44fc-9c12-de093c4a1510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Oct 30 02:29:21.830697 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:21.830651 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:21.968423 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:21.968390 5998 patch_prober.go:29] interesting pod/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Oct 30 02:29:21.968602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:21.968454 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" podUID="49169ae5-7185-44fc-9c12-de093c4a1510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Oct 30 02:29:22.968534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:22.968493 5998 patch_prober.go:29] interesting pod/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Oct 30 02:29:22.969072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:22.968560 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" podUID="49169ae5-7185-44fc-9c12-de093c4a1510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Oct 30 02:29:23.418175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:23.411305 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:23.967930 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:23.967889 5998 patch_prober.go:29] interesting pod/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Oct 30 02:29:23.968132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:23.967947 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" podUID="49169ae5-7185-44fc-9c12-de093c4a1510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Oct 30 02:29:25.091510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:25.091454 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:29:25.094191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:25.094162 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:29:25.097776 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:25.097753 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:29:25.119224 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:25.119183 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:25.237426 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:29:25.237390 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:29:26.666800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:26.666772 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:28.293893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:28.293856 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:29.873184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:29.873150 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:30.596114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:30.595429 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:31.581373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:31.581331 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:33.041238 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:33.041198 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc"] Oct 30 02:29:33.054839 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:33.054803 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-644a4b02-hl9cc"] Oct 30 02:29:33.441261 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:33.441219 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:33.778071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:33.767676 5998 scope.go:122] "RemoveContainer" containerID="fb919ea9b020bb5e950dfcdac79b7548ac6f84c624ca9dce0fa45327304132bd" Oct 30 02:29:33.831387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:33.831362 5998 scope.go:122] "RemoveContainer" containerID="9b89f8190f820871e320dacda823224fef4cb21d353d0af9d6544350e0773f16" Oct 30 02:29:33.843393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:33.843372 5998 scope.go:122] "RemoveContainer" containerID="13a8035afaff1aba96ff537059f0b0f2d24242ec91e791fda052bfa10d0d5e5d" Oct 30 02:29:34.161147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:34.161064 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:34.573502 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:34.573417 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3b76a87-372a-4147-9b84-f1ef6048f0b3" path="/var/lib/kubelet/pods/a3b76a87-372a-4147-9b84-f1ef6048f0b3/volumes" Oct 30 02:29:35.395316 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:35.395278 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:37.010553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:37.010515 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:38.689252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:38.689215 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:39.412646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:39.412613 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:40.360060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:40.360025 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:41.030557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:41.030520 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:42.086596 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:42.086447 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:43.708604 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:43.708562 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:44.034166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:44.033991 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c"] Oct 30 02:29:44.042775 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:44.042740 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-799fdeb2-mhj6c"] Oct 30 02:29:44.542897 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:44.542849 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcbc3c28-015c-4c5d-8c02-02ffb6becda3" path="/var/lib/kubelet/pods/dcbc3c28-015c-4c5d-8c02-02ffb6becda3/volumes" Oct 30 02:29:45.302196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:45.302162 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:46.967292 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:46.967254 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:48.058458 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:48.058418 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:48.672351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:48.672313 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:50.205804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:50.205759 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:51.789498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:51.789460 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:53.296868 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.296836 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:53.703820 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.703778 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf"] Oct 30 02:29:53.710774 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.710746 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:53.733754 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.733723 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf"] Oct 30 02:29:53.734077 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.734048 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-db01c4\"/\"openshift-service-ca.crt\"" Oct 30 02:29:53.734292 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.734274 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-db01c4\"/\"kube-root-ca.crt\"" Oct 30 02:29:53.749437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.740998 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-db01c4\"/\"default-dockercfg-6n72z\"" Oct 30 02:29:53.749437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.741129 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-db01c4\"/\"test-model-db01c4-kserve-self-signed-certs\"" Oct 30 02:29:53.760837 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.760807 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:53.833717 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.833677 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-dshm\") pod \"test-model-db01c4-kserve-786876d559-w8cbf\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:53.834082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.834057 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-tmp-dir\") pod \"test-model-db01c4-kserve-786876d559-w8cbf\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:53.834293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.834275 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-home\") pod \"test-model-db01c4-kserve-786876d559-w8cbf\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:53.834425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.834409 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3524261-ac9b-419e-82b9-c46404256f89-tls-certs\") pod \"test-model-db01c4-kserve-786876d559-w8cbf\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:53.834648 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.834631 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr6dx\" (UniqueName: \"kubernetes.io/projected/c3524261-ac9b-419e-82b9-c46404256f89-kube-api-access-zr6dx\") pod \"test-model-db01c4-kserve-786876d559-w8cbf\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:53.834778 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.834761 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-model-cache\") pod \"test-model-db01c4-kserve-786876d559-w8cbf\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:53.948484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.948282 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-dshm\") pod \"test-model-db01c4-kserve-786876d559-w8cbf\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:53.958618 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.958514 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-tmp-dir\") pod \"test-model-db01c4-kserve-786876d559-w8cbf\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:53.958898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.958154 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-tmp-dir\") pod \"test-model-db01c4-kserve-786876d559-w8cbf\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:53.959200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.959181 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-home\") pod \"test-model-db01c4-kserve-786876d559-w8cbf\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:53.963675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.963645 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3524261-ac9b-419e-82b9-c46404256f89-tls-certs\") pod \"test-model-db01c4-kserve-786876d559-w8cbf\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:53.967578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.963572 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-home\") pod \"test-model-db01c4-kserve-786876d559-w8cbf\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:53.984346 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.973636 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-dshm\") pod \"test-model-db01c4-kserve-786876d559-w8cbf\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:53.985272 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.985151 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr6dx\" (UniqueName: \"kubernetes.io/projected/c3524261-ac9b-419e-82b9-c46404256f89-kube-api-access-zr6dx\") pod \"test-model-db01c4-kserve-786876d559-w8cbf\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:53.985506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.985478 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-model-cache\") pod \"test-model-db01c4-kserve-786876d559-w8cbf\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:53.986190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:53.985956 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-model-cache\") pod \"test-model-db01c4-kserve-786876d559-w8cbf\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:54.027972 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:54.027930 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3524261-ac9b-419e-82b9-c46404256f89-tls-certs\") pod \"test-model-db01c4-kserve-786876d559-w8cbf\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:54.035120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:54.034983 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr6dx\" (UniqueName: \"kubernetes.io/projected/c3524261-ac9b-419e-82b9-c46404256f89-kube-api-access-zr6dx\") pod \"test-model-db01c4-kserve-786876d559-w8cbf\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:54.330744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:54.330641 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:54.651781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:54.651745 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf"] Oct 30 02:29:54.653175 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:29:54.653145 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3524261_ac9b_419e_82b9_c46404256f89.slice/crio-8a7901b9bcffef25a9127d72a6cc3227f44cccb7830ea09cef17a20fdc406ce5 WatchSource:0}: Error finding container 8a7901b9bcffef25a9127d72a6cc3227f44cccb7830ea09cef17a20fdc406ce5: Status 404 returned error can't find the container with id 8a7901b9bcffef25a9127d72a6cc3227f44cccb7830ea09cef17a20fdc406ce5 Oct 30 02:29:54.700253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:54.700216 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" event={"ID":"c3524261-ac9b-419e-82b9-c46404256f89","Type":"ContainerStarted","Data":"8a7901b9bcffef25a9127d72a6cc3227f44cccb7830ea09cef17a20fdc406ce5"} Oct 30 02:29:54.963426 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:54.963388 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:55.223238 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:29:55.223157 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:29:55.710929 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:55.710882 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" event={"ID":"c3524261-ac9b-419e-82b9-c46404256f89","Type":"ContainerStarted","Data":"f6168882745d2128ba538e8bb0788437761afbdfa8351e3e3512edfd71044d0b"} Oct 30 02:29:55.753493 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:55.753439 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" podStartSLOduration=2.753423831 podStartE2EDuration="2.753423831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:29:55.748905174 +0000 UTC m=+3625.327696255" watchObservedRunningTime="2026-10-30 02:29:55.753423831 +0000 UTC m=+3625.332214912" Oct 30 02:29:56.331668 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.331626 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:29:56.486697 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.486658 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t"] Oct 30 02:29:56.491773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.491736 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.504485 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.503736 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:56.527184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.506072 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d207e8\"/\"default-dockercfg-sbzzw\"" Oct 30 02:29:56.527184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.518436 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d207e8\"/\"test-model-d207e8-kserve-self-signed-certs\"" Oct 30 02:29:56.527184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.523521 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d207e8\"/\"openshift-service-ca.crt\"" Oct 30 02:29:56.527184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.523687 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d207e8\"/\"kube-root-ca.crt\"" Oct 30 02:29:56.527184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.526939 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t"] Oct 30 02:29:56.609676 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.609590 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:56.619944 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.619910 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrc42\" (UniqueName: \"kubernetes.io/projected/f2daa657-48ed-40a5-b5ab-3e1596a707ed-kube-api-access-xrc42\") pod \"test-model-d207e8-kserve-5dfc9fc475-nbx4t\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.620163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.620135 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-tmp-dir\") pod \"test-model-d207e8-kserve-5dfc9fc475-nbx4t\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.620244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.620180 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-home\") pod \"test-model-d207e8-kserve-5dfc9fc475-nbx4t\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.620244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.620212 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2daa657-48ed-40a5-b5ab-3e1596a707ed-tls-certs\") pod \"test-model-d207e8-kserve-5dfc9fc475-nbx4t\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.620356 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.620264 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-dshm\") pod \"test-model-d207e8-kserve-5dfc9fc475-nbx4t\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.620425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.620355 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-model-cache\") pod \"test-model-d207e8-kserve-5dfc9fc475-nbx4t\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.722192 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.722156 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrc42\" (UniqueName: \"kubernetes.io/projected/f2daa657-48ed-40a5-b5ab-3e1596a707ed-kube-api-access-xrc42\") pod \"test-model-d207e8-kserve-5dfc9fc475-nbx4t\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.722895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.722864 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-tmp-dir\") pod \"test-model-d207e8-kserve-5dfc9fc475-nbx4t\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.723044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.722906 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-home\") pod \"test-model-d207e8-kserve-5dfc9fc475-nbx4t\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.723044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.722933 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2daa657-48ed-40a5-b5ab-3e1596a707ed-tls-certs\") pod \"test-model-d207e8-kserve-5dfc9fc475-nbx4t\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.723044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.722968 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-dshm\") pod \"test-model-d207e8-kserve-5dfc9fc475-nbx4t\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.723274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.723084 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-model-cache\") pod \"test-model-d207e8-kserve-5dfc9fc475-nbx4t\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.723387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.723361 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-home\") pod \"test-model-d207e8-kserve-5dfc9fc475-nbx4t\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.723817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.723790 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-tmp-dir\") pod \"test-model-d207e8-kserve-5dfc9fc475-nbx4t\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.723997 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.723975 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-model-cache\") pod \"test-model-d207e8-kserve-5dfc9fc475-nbx4t\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.726052 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.726016 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2daa657-48ed-40a5-b5ab-3e1596a707ed-tls-certs\") pod \"test-model-d207e8-kserve-5dfc9fc475-nbx4t\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.726154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.726085 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-dshm\") pod \"test-model-d207e8-kserve-5dfc9fc475-nbx4t\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.744058 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.744023 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrc42\" (UniqueName: \"kubernetes.io/projected/f2daa657-48ed-40a5-b5ab-3e1596a707ed-kube-api-access-xrc42\") pod \"test-model-d207e8-kserve-5dfc9fc475-nbx4t\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:56.818126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:56.810231 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:57.319422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:57.319378 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t"] Oct 30 02:29:57.320201 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:29:57.320167 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2daa657_48ed_40a5_b5ab_3e1596a707ed.slice/crio-3d034068ce3432d64012a13d9ae814e201fbc123cf32f5280382fe1d2b42d155 WatchSource:0}: Error finding container 3d034068ce3432d64012a13d9ae814e201fbc123cf32f5280382fe1d2b42d155: Status 404 returned error can't find the container with id 3d034068ce3432d64012a13d9ae814e201fbc123cf32f5280382fe1d2b42d155 Oct 30 02:29:57.735063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:57.735023 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" event={"ID":"f2daa657-48ed-40a5-b5ab-3e1596a707ed","Type":"ContainerStarted","Data":"f1bc8eb987126f4f8c19eb637e88ccc35af9e2a29ca69c631debc7a2ae7390ac"} Oct 30 02:29:57.735565 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:57.735072 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" event={"ID":"f2daa657-48ed-40a5-b5ab-3e1596a707ed","Type":"ContainerStarted","Data":"3d034068ce3432d64012a13d9ae814e201fbc123cf32f5280382fe1d2b42d155"} Oct 30 02:29:57.776840 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:57.776787 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" podStartSLOduration=1.776772032 podStartE2EDuration="1.776772032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:29:57.776144004 +0000 UTC m=+3627.354935085" watchObservedRunningTime="2026-10-30 02:29:57.776772032 +0000 UTC m=+3627.355563113" Oct 30 02:29:57.811570 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:57.811533 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:29:58.088605 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:58.085571 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:29:59.507113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:59.506678 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:29:59.893592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:29:59.893505 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:00.153780 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.153696 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw"] Oct 30 02:30:00.158799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.158766 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6"] Oct 30 02:30:00.159020 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.158997 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" Oct 30 02:30:00.163754 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.163727 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq"] Oct 30 02:30:00.164027 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.163817 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" Oct 30 02:30:00.168818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.168792 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" Oct 30 02:30:00.169533 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.169423 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh"] Oct 30 02:30:00.174660 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.174633 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" Oct 30 02:30:00.174965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.174944 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z"] Oct 30 02:30:00.179511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.179492 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" Oct 30 02:30:00.180379 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.180349 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:30:00.180379 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.180357 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:30:00.190889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.190839 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:30:00.192260 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.191762 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw"] Oct 30 02:30:00.197345 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.197323 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd"] Oct 30 02:30:00.261049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.259035 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5882ea9-fe2e-4587-abb7-bb14939f53f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6\" (UID: \"e5882ea9-fe2e-4587-abb7-bb14939f53f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" Oct 30 02:30:00.261049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.259146 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f69dd9b-8eda-4368-b693-33c5b9721710-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh\" (UID: \"3f69dd9b-8eda-4368-b693-33c5b9721710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" Oct 30 02:30:00.261049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.259198 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2hls\" (UniqueName: \"kubernetes.io/projected/01d4f72c-dc27-4988-b301-873c276855bd-kube-api-access-z2hls\") pod \"maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw\" (UID: \"01d4f72c-dc27-4988-b301-873c276855bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" Oct 30 02:30:00.261049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.259255 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn4s7\" (UniqueName: \"kubernetes.io/projected/e5882ea9-fe2e-4587-abb7-bb14939f53f6-kube-api-access-jn4s7\") pod \"maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6\" (UID: \"e5882ea9-fe2e-4587-abb7-bb14939f53f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" Oct 30 02:30:00.261049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.259304 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8qj6\" (UniqueName: \"kubernetes.io/projected/8e81c096-2704-4d9e-80b8-9b1f05e7151f-kube-api-access-m8qj6\") pod \"maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z\" (UID: \"8e81c096-2704-4d9e-80b8-9b1f05e7151f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" Oct 30 02:30:00.261049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.259329 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k44z7\" (UniqueName: \"kubernetes.io/projected/512be8ad-51e7-4919-a456-9fcfafab46bd-kube-api-access-k44z7\") pod \"maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq\" (UID: \"512be8ad-51e7-4919-a456-9fcfafab46bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" Oct 30 02:30:00.261049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.259398 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d4f72c-dc27-4988-b301-873c276855bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw\" (UID: \"01d4f72c-dc27-4988-b301-873c276855bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" Oct 30 02:30:00.261049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.259447 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh5lf\" (UniqueName: \"kubernetes.io/projected/3f69dd9b-8eda-4368-b693-33c5b9721710-kube-api-access-hh5lf\") pod \"maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh\" (UID: \"3f69dd9b-8eda-4368-b693-33c5b9721710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" Oct 30 02:30:00.261049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.259528 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e81c096-2704-4d9e-80b8-9b1f05e7151f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z\" (UID: \"8e81c096-2704-4d9e-80b8-9b1f05e7151f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" Oct 30 02:30:00.261049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.259557 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512be8ad-51e7-4919-a456-9fcfafab46bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq\" (UID: \"512be8ad-51e7-4919-a456-9fcfafab46bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" Oct 30 02:30:00.261049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.260903 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" Oct 30 02:30:00.261806 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.261082 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf"] Oct 30 02:30:00.285160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.280052 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6"] Oct 30 02:30:00.285160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.280164 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp"] Oct 30 02:30:00.285160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.283755 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" Oct 30 02:30:00.312019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.311555 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp"] Oct 30 02:30:00.312019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.311959 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" Oct 30 02:30:00.325332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.322301 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn"] Oct 30 02:30:00.325332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.322932 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.362535 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4s7\" (UniqueName: \"kubernetes.io/projected/e5882ea9-fe2e-4587-abb7-bb14939f53f6-kube-api-access-jn4s7\") pod \"maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6\" (UID: \"e5882ea9-fe2e-4587-abb7-bb14939f53f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.362602 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8qj6\" (UniqueName: \"kubernetes.io/projected/8e81c096-2704-4d9e-80b8-9b1f05e7151f-kube-api-access-m8qj6\") pod \"maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z\" (UID: \"8e81c096-2704-4d9e-80b8-9b1f05e7151f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.362632 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k44z7\" (UniqueName: \"kubernetes.io/projected/512be8ad-51e7-4919-a456-9fcfafab46bd-kube-api-access-k44z7\") pod \"maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq\" (UID: \"512be8ad-51e7-4919-a456-9fcfafab46bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.362700 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n8xl\" (UniqueName: \"kubernetes.io/projected/7ff76844-137e-44ec-b146-4d9c84997ada-kube-api-access-2n8xl\") pod \"maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp\" (UID: \"7ff76844-137e-44ec-b146-4d9c84997ada\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.362776 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d4f72c-dc27-4988-b301-873c276855bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw\" (UID: \"01d4f72c-dc27-4988-b301-873c276855bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.362831 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svhbs\" (UniqueName: \"kubernetes.io/projected/a1f42aec-2169-46f1-a837-039b920ac828-kube-api-access-svhbs\") pod \"maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd\" (UID: \"a1f42aec-2169-46f1-a837-039b920ac828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.362866 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh5lf\" (UniqueName: \"kubernetes.io/projected/3f69dd9b-8eda-4368-b693-33c5b9721710-kube-api-access-hh5lf\") pod \"maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh\" (UID: \"3f69dd9b-8eda-4368-b693-33c5b9721710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.362942 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77n7p\" (UniqueName: \"kubernetes.io/projected/be309ab3-73d7-42c0-92a8-0fc686504795-kube-api-access-77n7p\") pod \"maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf\" (UID: \"be309ab3-73d7-42c0-92a8-0fc686504795\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.362978 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f42aec-2169-46f1-a837-039b920ac828-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd\" (UID: \"a1f42aec-2169-46f1-a837-039b920ac828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.363006 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e81c096-2704-4d9e-80b8-9b1f05e7151f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z\" (UID: \"8e81c096-2704-4d9e-80b8-9b1f05e7151f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.363035 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512be8ad-51e7-4919-a456-9fcfafab46bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq\" (UID: \"512be8ad-51e7-4919-a456-9fcfafab46bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.363155 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkcpq\" (UniqueName: \"kubernetes.io/projected/c9b0ea41-0e61-456d-bbce-6ee7145552ea-kube-api-access-jkcpq\") pod \"maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp\" (UID: \"c9b0ea41-0e61-456d-bbce-6ee7145552ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.363182 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b0ea41-0e61-456d-bbce-6ee7145552ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp\" (UID: \"c9b0ea41-0e61-456d-bbce-6ee7145552ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.363212 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff76844-137e-44ec-b146-4d9c84997ada-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp\" (UID: \"7ff76844-137e-44ec-b146-4d9c84997ada\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.363240 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be309ab3-73d7-42c0-92a8-0fc686504795-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf\" (UID: \"be309ab3-73d7-42c0-92a8-0fc686504795\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.363308 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5882ea9-fe2e-4587-abb7-bb14939f53f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6\" (UID: \"e5882ea9-fe2e-4587-abb7-bb14939f53f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.363400 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f69dd9b-8eda-4368-b693-33c5b9721710-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh\" (UID: \"3f69dd9b-8eda-4368-b693-33c5b9721710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.363453 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2hls\" (UniqueName: \"kubernetes.io/projected/01d4f72c-dc27-4988-b301-873c276855bd-kube-api-access-z2hls\") pod \"maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw\" (UID: \"01d4f72c-dc27-4988-b301-873c276855bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.366116 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d4f72c-dc27-4988-b301-873c276855bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw\" (UID: \"01d4f72c-dc27-4988-b301-873c276855bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.367435 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e81c096-2704-4d9e-80b8-9b1f05e7151f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z\" (UID: \"8e81c096-2704-4d9e-80b8-9b1f05e7151f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.368058 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512be8ad-51e7-4919-a456-9fcfafab46bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq\" (UID: \"512be8ad-51e7-4919-a456-9fcfafab46bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.368706 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5882ea9-fe2e-4587-abb7-bb14939f53f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6\" (UID: \"e5882ea9-fe2e-4587-abb7-bb14939f53f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" Oct 30 02:30:00.372114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.369301 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f69dd9b-8eda-4368-b693-33c5b9721710-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh\" (UID: \"3f69dd9b-8eda-4368-b693-33c5b9721710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" Oct 30 02:30:00.385118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.374543 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh"] Oct 30 02:30:00.385118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.374609 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z"] Oct 30 02:30:00.385118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.374640 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq"] Oct 30 02:30:00.385118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.374776 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" Oct 30 02:30:00.395240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.389736 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g"] Oct 30 02:30:00.418059 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.416419 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw"] Oct 30 02:30:00.421258 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.420806 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh5lf\" (UniqueName: \"kubernetes.io/projected/3f69dd9b-8eda-4368-b693-33c5b9721710-kube-api-access-hh5lf\") pod \"maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh\" (UID: \"3f69dd9b-8eda-4368-b693-33c5b9721710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" Oct 30 02:30:00.421258 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.420943 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" Oct 30 02:30:00.425475 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.422481 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" Oct 30 02:30:00.426672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.426115 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2hls\" (UniqueName: \"kubernetes.io/projected/01d4f72c-dc27-4988-b301-873c276855bd-kube-api-access-z2hls\") pod \"maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw\" (UID: \"01d4f72c-dc27-4988-b301-873c276855bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" Oct 30 02:30:00.428136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.427656 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn4s7\" (UniqueName: \"kubernetes.io/projected/e5882ea9-fe2e-4587-abb7-bb14939f53f6-kube-api-access-jn4s7\") pod \"maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6\" (UID: \"e5882ea9-fe2e-4587-abb7-bb14939f53f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" Oct 30 02:30:00.429178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.429150 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc"] Oct 30 02:30:00.430614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.430571 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8qj6\" (UniqueName: \"kubernetes.io/projected/8e81c096-2704-4d9e-80b8-9b1f05e7151f-kube-api-access-m8qj6\") pod \"maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z\" (UID: \"8e81c096-2704-4d9e-80b8-9b1f05e7151f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" Oct 30 02:30:00.441211 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.433637 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k44z7\" (UniqueName: \"kubernetes.io/projected/512be8ad-51e7-4919-a456-9fcfafab46bd-kube-api-access-k44z7\") pod \"maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq\" (UID: \"512be8ad-51e7-4919-a456-9fcfafab46bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" Oct 30 02:30:00.463363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.462607 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc" Oct 30 02:30:00.471508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.470173 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2"] Oct 30 02:30:00.475477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.475053 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/860ba768-1529-403e-ba81-13395d6cee83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g\" (UID: \"860ba768-1529-403e-ba81-13395d6cee83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" Oct 30 02:30:00.475477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.475145 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw\" (UID: \"1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" Oct 30 02:30:00.475477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.475210 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg6kp\" (UniqueName: \"kubernetes.io/projected/1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee-kube-api-access-wg6kp\") pod \"maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw\" (UID: \"1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" Oct 30 02:30:00.475477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.475271 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4lln\" (UniqueName: \"kubernetes.io/projected/860ba768-1529-403e-ba81-13395d6cee83-kube-api-access-n4lln\") pod \"maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g\" (UID: \"860ba768-1529-403e-ba81-13395d6cee83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" Oct 30 02:30:00.475477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.475359 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n8xl\" (UniqueName: \"kubernetes.io/projected/7ff76844-137e-44ec-b146-4d9c84997ada-kube-api-access-2n8xl\") pod \"maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp\" (UID: \"7ff76844-137e-44ec-b146-4d9c84997ada\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" Oct 30 02:30:00.475477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.475410 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df3039d-5102-404f-b58f-eae1e79feb36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc\" (UID: \"6df3039d-5102-404f-b58f-eae1e79feb36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc" Oct 30 02:30:00.475477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.475453 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svhbs\" (UniqueName: \"kubernetes.io/projected/a1f42aec-2169-46f1-a837-039b920ac828-kube-api-access-svhbs\") pod \"maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd\" (UID: \"a1f42aec-2169-46f1-a837-039b920ac828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" Oct 30 02:30:00.475477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.475480 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vns4\" (UniqueName: \"kubernetes.io/projected/046ebf9b-3d7e-4c67-bfcf-8545980b59ff-kube-api-access-7vns4\") pod \"maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn\" (UID: \"046ebf9b-3d7e-4c67-bfcf-8545980b59ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" Oct 30 02:30:00.475999 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.475868 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77n7p\" (UniqueName: \"kubernetes.io/projected/be309ab3-73d7-42c0-92a8-0fc686504795-kube-api-access-77n7p\") pod \"maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf\" (UID: \"be309ab3-73d7-42c0-92a8-0fc686504795\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" Oct 30 02:30:00.475999 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.475912 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f42aec-2169-46f1-a837-039b920ac828-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd\" (UID: \"a1f42aec-2169-46f1-a837-039b920ac828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" Oct 30 02:30:00.475999 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.475964 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph497\" (UniqueName: \"kubernetes.io/projected/6df3039d-5102-404f-b58f-eae1e79feb36-kube-api-access-ph497\") pod \"maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc\" (UID: \"6df3039d-5102-404f-b58f-eae1e79feb36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc" Oct 30 02:30:00.476190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.476069 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs"] Oct 30 02:30:00.476190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.476106 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkcpq\" (UniqueName: \"kubernetes.io/projected/c9b0ea41-0e61-456d-bbce-6ee7145552ea-kube-api-access-jkcpq\") pod \"maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp\" (UID: \"c9b0ea41-0e61-456d-bbce-6ee7145552ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" Oct 30 02:30:00.476190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.476160 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b0ea41-0e61-456d-bbce-6ee7145552ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp\" (UID: \"c9b0ea41-0e61-456d-bbce-6ee7145552ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" Oct 30 02:30:00.476419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.476198 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff76844-137e-44ec-b146-4d9c84997ada-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp\" (UID: \"7ff76844-137e-44ec-b146-4d9c84997ada\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" Oct 30 02:30:00.476419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.476249 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be309ab3-73d7-42c0-92a8-0fc686504795-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf\" (UID: \"be309ab3-73d7-42c0-92a8-0fc686504795\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" Oct 30 02:30:00.476419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.476355 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046ebf9b-3d7e-4c67-bfcf-8545980b59ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn\" (UID: \"046ebf9b-3d7e-4c67-bfcf-8545980b59ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" Oct 30 02:30:00.477032 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.477007 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff76844-137e-44ec-b146-4d9c84997ada-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp\" (UID: \"7ff76844-137e-44ec-b146-4d9c84997ada\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" Oct 30 02:30:00.477605 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.477581 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f42aec-2169-46f1-a837-039b920ac828-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd\" (UID: \"a1f42aec-2169-46f1-a837-039b920ac828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" Oct 30 02:30:00.477767 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.477728 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be309ab3-73d7-42c0-92a8-0fc686504795-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf\" (UID: \"be309ab3-73d7-42c0-92a8-0fc686504795\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" Oct 30 02:30:00.478223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.478202 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" Oct 30 02:30:00.479051 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.479028 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" Oct 30 02:30:00.479371 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.479350 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b0ea41-0e61-456d-bbce-6ee7145552ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp\" (UID: \"c9b0ea41-0e61-456d-bbce-6ee7145552ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" Oct 30 02:30:00.486397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.484942 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd"] Oct 30 02:30:00.486397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.484981 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp"] Oct 30 02:30:00.486397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.485064 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" Oct 30 02:30:00.490144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.489030 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf"] Oct 30 02:30:00.494019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.493995 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp"] Oct 30 02:30:00.497134 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.497079 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" Oct 30 02:30:00.500252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.499152 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k"] Oct 30 02:30:00.500252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.499480 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkcpq\" (UniqueName: \"kubernetes.io/projected/c9b0ea41-0e61-456d-bbce-6ee7145552ea-kube-api-access-jkcpq\") pod \"maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp\" (UID: \"c9b0ea41-0e61-456d-bbce-6ee7145552ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" Oct 30 02:30:00.500252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.499599 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77n7p\" (UniqueName: \"kubernetes.io/projected/be309ab3-73d7-42c0-92a8-0fc686504795-kube-api-access-77n7p\") pod \"maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf\" (UID: \"be309ab3-73d7-42c0-92a8-0fc686504795\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" Oct 30 02:30:00.500252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.499653 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n8xl\" (UniqueName: \"kubernetes.io/projected/7ff76844-137e-44ec-b146-4d9c84997ada-kube-api-access-2n8xl\") pod \"maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp\" (UID: \"7ff76844-137e-44ec-b146-4d9c84997ada\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" Oct 30 02:30:00.502300 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.502278 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svhbs\" (UniqueName: \"kubernetes.io/projected/a1f42aec-2169-46f1-a837-039b920ac828-kube-api-access-svhbs\") pod \"maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd\" (UID: \"a1f42aec-2169-46f1-a837-039b920ac828\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" Oct 30 02:30:00.504783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.504647 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" Oct 30 02:30:00.505764 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.505746 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn"] Oct 30 02:30:00.518373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.518209 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt"] Oct 30 02:30:00.519816 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.518339 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" Oct 30 02:30:00.530136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.528299 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn"] Oct 30 02:30:00.530136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.528346 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g"] Oct 30 02:30:00.530136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.528448 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" Oct 30 02:30:00.535159 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.534578 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" Oct 30 02:30:00.565052 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.559105 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc"] Oct 30 02:30:00.565052 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.559168 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2"] Oct 30 02:30:00.565052 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.559205 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr"] Oct 30 02:30:00.581818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.580612 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046ebf9b-3d7e-4c67-bfcf-8545980b59ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn\" (UID: \"046ebf9b-3d7e-4c67-bfcf-8545980b59ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" Oct 30 02:30:00.581818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.580652 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a105635a-7115-4c02-aa79-60516391579b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt\" (UID: \"a105635a-7115-4c02-aa79-60516391579b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" Oct 30 02:30:00.581818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.580698 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daaa2b88-a49a-4a88-8325-33307e969074-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2\" (UID: \"daaa2b88-a49a-4a88-8325-33307e969074\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" Oct 30 02:30:00.581818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.580739 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/860ba768-1529-403e-ba81-13395d6cee83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g\" (UID: \"860ba768-1529-403e-ba81-13395d6cee83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" Oct 30 02:30:00.581818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.580759 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw\" (UID: \"1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" Oct 30 02:30:00.581818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.580816 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg6kp\" (UniqueName: \"kubernetes.io/projected/1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee-kube-api-access-wg6kp\") pod \"maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw\" (UID: \"1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" Oct 30 02:30:00.581818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.580841 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbr5n\" (UniqueName: \"kubernetes.io/projected/5bce6d68-142e-46ef-b408-89e44ca0fc10-kube-api-access-gbr5n\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k\" (UID: \"5bce6d68-142e-46ef-b408-89e44ca0fc10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" Oct 30 02:30:00.581818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.580858 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn\" (UID: \"3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" Oct 30 02:30:00.581818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.580895 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxtdv\" (UniqueName: \"kubernetes.io/projected/a105635a-7115-4c02-aa79-60516391579b-kube-api-access-pxtdv\") pod \"maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt\" (UID: \"a105635a-7115-4c02-aa79-60516391579b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" Oct 30 02:30:00.581818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.580919 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4lln\" (UniqueName: \"kubernetes.io/projected/860ba768-1529-403e-ba81-13395d6cee83-kube-api-access-n4lln\") pod \"maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g\" (UID: \"860ba768-1529-403e-ba81-13395d6cee83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" Oct 30 02:30:00.581818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.580967 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df3039d-5102-404f-b58f-eae1e79feb36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc\" (UID: \"6df3039d-5102-404f-b58f-eae1e79feb36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc" Oct 30 02:30:00.581818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.580986 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bce6d68-142e-46ef-b408-89e44ca0fc10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k\" (UID: \"5bce6d68-142e-46ef-b408-89e44ca0fc10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" Oct 30 02:30:00.581818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.581002 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjdxg\" (UniqueName: \"kubernetes.io/projected/2f3ead60-88ca-42af-9e25-7822589058e4-kube-api-access-gjdxg\") pod \"maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs\" (UID: \"2f3ead60-88ca-42af-9e25-7822589058e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" Oct 30 02:30:00.581818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.581031 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vns4\" (UniqueName: \"kubernetes.io/projected/046ebf9b-3d7e-4c67-bfcf-8545980b59ff-kube-api-access-7vns4\") pod \"maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn\" (UID: \"046ebf9b-3d7e-4c67-bfcf-8545980b59ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" Oct 30 02:30:00.581818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.581053 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3ead60-88ca-42af-9e25-7822589058e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs\" (UID: \"2f3ead60-88ca-42af-9e25-7822589058e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" Oct 30 02:30:00.581818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.581071 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvcsj\" (UniqueName: \"kubernetes.io/projected/3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb-kube-api-access-wvcsj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn\" (UID: \"3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" Oct 30 02:30:00.583005 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.581901 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046ebf9b-3d7e-4c67-bfcf-8545980b59ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn\" (UID: \"046ebf9b-3d7e-4c67-bfcf-8545980b59ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" Oct 30 02:30:00.583005 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.582664 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/860ba768-1529-403e-ba81-13395d6cee83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g\" (UID: \"860ba768-1529-403e-ba81-13395d6cee83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" Oct 30 02:30:00.584451 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.584265 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw\" (UID: \"1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" Oct 30 02:30:00.584589 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.584413 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmtcf\" (UniqueName: \"kubernetes.io/projected/daaa2b88-a49a-4a88-8325-33307e969074-kube-api-access-cmtcf\") pod \"maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2\" (UID: \"daaa2b88-a49a-4a88-8325-33307e969074\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" Oct 30 02:30:00.584820 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.584773 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph497\" (UniqueName: \"kubernetes.io/projected/6df3039d-5102-404f-b58f-eae1e79feb36-kube-api-access-ph497\") pod \"maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc\" (UID: \"6df3039d-5102-404f-b58f-eae1e79feb36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc" Oct 30 02:30:00.585567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.585499 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df3039d-5102-404f-b58f-eae1e79feb36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc\" (UID: \"6df3039d-5102-404f-b58f-eae1e79feb36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc" Oct 30 02:30:00.586675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.586651 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k"] Oct 30 02:30:00.586844 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.586830 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj"] Oct 30 02:30:00.587257 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.586856 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" Oct 30 02:30:00.587970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.586761 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" Oct 30 02:30:00.598373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.598150 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" Oct 30 02:30:00.616698 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.616667 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr"] Oct 30 02:30:00.616996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.616975 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw"] Oct 30 02:30:00.618215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.617156 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs"] Oct 30 02:30:00.619531 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.619364 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" Oct 30 02:30:00.636418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.636381 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" Oct 30 02:30:00.651749 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.651720 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4lln\" (UniqueName: \"kubernetes.io/projected/860ba768-1529-403e-ba81-13395d6cee83-kube-api-access-n4lln\") pod \"maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g\" (UID: \"860ba768-1529-403e-ba81-13395d6cee83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" Oct 30 02:30:00.660790 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.660106 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vns4\" (UniqueName: \"kubernetes.io/projected/046ebf9b-3d7e-4c67-bfcf-8545980b59ff-kube-api-access-7vns4\") pod \"maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn\" (UID: \"046ebf9b-3d7e-4c67-bfcf-8545980b59ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" Oct 30 02:30:00.661288 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.661232 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg6kp\" (UniqueName: \"kubernetes.io/projected/1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee-kube-api-access-wg6kp\") pod \"maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw\" (UID: \"1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" Oct 30 02:30:00.669220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.668967 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" Oct 30 02:30:00.675808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.675781 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph497\" (UniqueName: \"kubernetes.io/projected/6df3039d-5102-404f-b58f-eae1e79feb36-kube-api-access-ph497\") pod \"maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc\" (UID: \"6df3039d-5102-404f-b58f-eae1e79feb36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc" Oct 30 02:30:00.675888 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.675841 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff"] Oct 30 02:30:00.686763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.686730 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bc88e1-7b35-4b54-b12f-69227ae41a88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj\" (UID: \"96bc88e1-7b35-4b54-b12f-69227ae41a88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" Oct 30 02:30:00.686901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.686786 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmtcf\" (UniqueName: \"kubernetes.io/projected/daaa2b88-a49a-4a88-8325-33307e969074-kube-api-access-cmtcf\") pod \"maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2\" (UID: \"daaa2b88-a49a-4a88-8325-33307e969074\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" Oct 30 02:30:00.686901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.686875 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" Oct 30 02:30:00.687019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.686950 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4cvr\" (UniqueName: \"kubernetes.io/projected/96bc88e1-7b35-4b54-b12f-69227ae41a88-kube-api-access-v4cvr\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj\" (UID: \"96bc88e1-7b35-4b54-b12f-69227ae41a88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" Oct 30 02:30:00.687121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.687101 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a105635a-7115-4c02-aa79-60516391579b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt\" (UID: \"a105635a-7115-4c02-aa79-60516391579b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" Oct 30 02:30:00.687229 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.687172 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daaa2b88-a49a-4a88-8325-33307e969074-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2\" (UID: \"daaa2b88-a49a-4a88-8325-33307e969074\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" Oct 30 02:30:00.687359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.687269 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbr5n\" (UniqueName: \"kubernetes.io/projected/5bce6d68-142e-46ef-b408-89e44ca0fc10-kube-api-access-gbr5n\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k\" (UID: \"5bce6d68-142e-46ef-b408-89e44ca0fc10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" Oct 30 02:30:00.687359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.687302 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn\" (UID: \"3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" Oct 30 02:30:00.687359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.687342 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxtdv\" (UniqueName: \"kubernetes.io/projected/a105635a-7115-4c02-aa79-60516391579b-kube-api-access-pxtdv\") pod \"maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt\" (UID: \"a105635a-7115-4c02-aa79-60516391579b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" Oct 30 02:30:00.687545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.687398 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512d72f9-4883-4d39-9df8-9282f56da2ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr\" (UID: \"512d72f9-4883-4d39-9df8-9282f56da2ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" Oct 30 02:30:00.687545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.687469 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bce6d68-142e-46ef-b408-89e44ca0fc10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k\" (UID: \"5bce6d68-142e-46ef-b408-89e44ca0fc10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" Oct 30 02:30:00.687545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.687501 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjdxg\" (UniqueName: \"kubernetes.io/projected/2f3ead60-88ca-42af-9e25-7822589058e4-kube-api-access-gjdxg\") pod \"maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs\" (UID: \"2f3ead60-88ca-42af-9e25-7822589058e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" Oct 30 02:30:00.687545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.687537 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq6kr\" (UniqueName: \"kubernetes.io/projected/512d72f9-4883-4d39-9df8-9282f56da2ac-kube-api-access-wq6kr\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr\" (UID: \"512d72f9-4883-4d39-9df8-9282f56da2ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" Oct 30 02:30:00.687776 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.687576 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3ead60-88ca-42af-9e25-7822589058e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs\" (UID: \"2f3ead60-88ca-42af-9e25-7822589058e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" Oct 30 02:30:00.687776 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.687603 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvcsj\" (UniqueName: \"kubernetes.io/projected/3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb-kube-api-access-wvcsj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn\" (UID: \"3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" Oct 30 02:30:00.691508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.688371 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a105635a-7115-4c02-aa79-60516391579b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt\" (UID: \"a105635a-7115-4c02-aa79-60516391579b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" Oct 30 02:30:00.691508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.688771 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn\" (UID: \"3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" Oct 30 02:30:00.691508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.689084 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daaa2b88-a49a-4a88-8325-33307e969074-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2\" (UID: \"daaa2b88-a49a-4a88-8325-33307e969074\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" Oct 30 02:30:00.691508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.689947 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3ead60-88ca-42af-9e25-7822589058e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs\" (UID: \"2f3ead60-88ca-42af-9e25-7822589058e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" Oct 30 02:30:00.691508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.690024 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bce6d68-142e-46ef-b408-89e44ca0fc10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k\" (UID: \"5bce6d68-142e-46ef-b408-89e44ca0fc10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" Oct 30 02:30:00.692665 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.692621 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" Oct 30 02:30:00.705656 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.705637 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn"] Oct 30 02:30:00.722067 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.722005 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvcsj\" (UniqueName: \"kubernetes.io/projected/3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb-kube-api-access-wvcsj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn\" (UID: \"3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" Oct 30 02:30:00.727389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.726644 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj"] Oct 30 02:30:00.727389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.726732 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt"] Oct 30 02:30:00.728801 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.728494 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmtcf\" (UniqueName: \"kubernetes.io/projected/daaa2b88-a49a-4a88-8325-33307e969074-kube-api-access-cmtcf\") pod \"maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2\" (UID: \"daaa2b88-a49a-4a88-8325-33307e969074\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" Oct 30 02:30:00.730863 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.730835 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" Oct 30 02:30:00.732019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.731966 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbr5n\" (UniqueName: \"kubernetes.io/projected/5bce6d68-142e-46ef-b408-89e44ca0fc10-kube-api-access-gbr5n\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k\" (UID: \"5bce6d68-142e-46ef-b408-89e44ca0fc10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" Oct 30 02:30:00.746435 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.735655 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc"] Oct 30 02:30:00.746435 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.742825 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf"] Oct 30 02:30:00.750682 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.750652 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" Oct 30 02:30:00.751051 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.751032 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" Oct 30 02:30:00.822601 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.776454 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxtdv\" (UniqueName: \"kubernetes.io/projected/a105635a-7115-4c02-aa79-60516391579b-kube-api-access-pxtdv\") pod \"maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt\" (UID: \"a105635a-7115-4c02-aa79-60516391579b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" Oct 30 02:30:00.822601 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.784024 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" Oct 30 02:30:00.849182 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.823361 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" Oct 30 02:30:00.904792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.849650 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" Oct 30 02:30:00.904792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.850080 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx4c2\" (UniqueName: \"kubernetes.io/projected/0771e950-74b9-45cb-a7be-ccb30c8c8c65-kube-api-access-dx4c2\") pod \"maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf\" (UID: \"0771e950-74b9-45cb-a7be-ccb30c8c8c65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" Oct 30 02:30:00.904792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.850167 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512d72f9-4883-4d39-9df8-9282f56da2ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr\" (UID: \"512d72f9-4883-4d39-9df8-9282f56da2ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" Oct 30 02:30:00.904792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.850245 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq6kr\" (UniqueName: \"kubernetes.io/projected/512d72f9-4883-4d39-9df8-9282f56da2ac-kube-api-access-wq6kr\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr\" (UID: \"512d72f9-4883-4d39-9df8-9282f56da2ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" Oct 30 02:30:00.904792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.850333 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bc88e1-7b35-4b54-b12f-69227ae41a88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj\" (UID: \"96bc88e1-7b35-4b54-b12f-69227ae41a88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" Oct 30 02:30:00.904792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.850355 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0771e950-74b9-45cb-a7be-ccb30c8c8c65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf\" (UID: \"0771e950-74b9-45cb-a7be-ccb30c8c8c65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" Oct 30 02:30:00.904792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.850451 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkvfw\" (UniqueName: \"kubernetes.io/projected/115cd7ff-80aa-4cd9-a247-9154fe7d117f-kube-api-access-vkvfw\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff\" (UID: \"115cd7ff-80aa-4cd9-a247-9154fe7d117f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" Oct 30 02:30:00.904792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.850483 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clr67\" (UniqueName: \"kubernetes.io/projected/9e4909d0-f3ae-423a-a478-4ee5ce9734a1-kube-api-access-clr67\") pod \"maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc\" (UID: \"9e4909d0-f3ae-423a-a478-4ee5ce9734a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" Oct 30 02:30:00.904792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.850562 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4cvr\" (UniqueName: \"kubernetes.io/projected/96bc88e1-7b35-4b54-b12f-69227ae41a88-kube-api-access-v4cvr\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj\" (UID: \"96bc88e1-7b35-4b54-b12f-69227ae41a88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" Oct 30 02:30:00.904792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.850705 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e4909d0-f3ae-423a-a478-4ee5ce9734a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc\" (UID: \"9e4909d0-f3ae-423a-a478-4ee5ce9734a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" Oct 30 02:30:00.904792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.850781 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/115cd7ff-80aa-4cd9-a247-9154fe7d117f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff\" (UID: \"115cd7ff-80aa-4cd9-a247-9154fe7d117f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" Oct 30 02:30:00.904792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.852066 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bc88e1-7b35-4b54-b12f-69227ae41a88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj\" (UID: \"96bc88e1-7b35-4b54-b12f-69227ae41a88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" Oct 30 02:30:00.904792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.852775 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512d72f9-4883-4d39-9df8-9282f56da2ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr\" (UID: \"512d72f9-4883-4d39-9df8-9282f56da2ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" Oct 30 02:30:00.939167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.912883 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" Oct 30 02:30:00.982926 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.982894 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" Oct 30 02:30:01.007270 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.984010 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e4909d0-f3ae-423a-a478-4ee5ce9734a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc\" (UID: \"9e4909d0-f3ae-423a-a478-4ee5ce9734a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" Oct 30 02:30:01.007526 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.967025 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e4909d0-f3ae-423a-a478-4ee5ce9734a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc\" (UID: \"9e4909d0-f3ae-423a-a478-4ee5ce9734a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" Oct 30 02:30:01.007739 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.007719 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/115cd7ff-80aa-4cd9-a247-9154fe7d117f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff\" (UID: \"115cd7ff-80aa-4cd9-a247-9154fe7d117f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" Oct 30 02:30:01.008032 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.008011 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx4c2\" (UniqueName: \"kubernetes.io/projected/0771e950-74b9-45cb-a7be-ccb30c8c8c65-kube-api-access-dx4c2\") pod \"maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf\" (UID: \"0771e950-74b9-45cb-a7be-ccb30c8c8c65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" Oct 30 02:30:01.042786 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.042748 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0771e950-74b9-45cb-a7be-ccb30c8c8c65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf\" (UID: \"0771e950-74b9-45cb-a7be-ccb30c8c8c65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" Oct 30 02:30:01.059119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:00.984277 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc" Oct 30 02:30:01.199167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.039656 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" Oct 30 02:30:01.199167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.039030 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/115cd7ff-80aa-4cd9-a247-9154fe7d117f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff\" (UID: \"115cd7ff-80aa-4cd9-a247-9154fe7d117f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" Oct 30 02:30:01.199167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.067317 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkvfw\" (UniqueName: \"kubernetes.io/projected/115cd7ff-80aa-4cd9-a247-9154fe7d117f-kube-api-access-vkvfw\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff\" (UID: \"115cd7ff-80aa-4cd9-a247-9154fe7d117f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" Oct 30 02:30:01.199167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.122770 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clr67\" (UniqueName: \"kubernetes.io/projected/9e4909d0-f3ae-423a-a478-4ee5ce9734a1-kube-api-access-clr67\") pod \"maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc\" (UID: \"9e4909d0-f3ae-423a-a478-4ee5ce9734a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" Oct 30 02:30:01.199167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.081845 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0771e950-74b9-45cb-a7be-ccb30c8c8c65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf\" (UID: \"0771e950-74b9-45cb-a7be-ccb30c8c8c65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" Oct 30 02:30:01.199167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.154601 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" Oct 30 02:30:01.285254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.221022 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjdxg\" (UniqueName: \"kubernetes.io/projected/2f3ead60-88ca-42af-9e25-7822589058e4-kube-api-access-gjdxg\") pod \"maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs\" (UID: \"2f3ead60-88ca-42af-9e25-7822589058e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" Oct 30 02:30:01.507610 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.502722 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff"] Oct 30 02:30:01.507995 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.507280 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx4c2\" (UniqueName: \"kubernetes.io/projected/0771e950-74b9-45cb-a7be-ccb30c8c8c65-kube-api-access-dx4c2\") pod \"maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf\" (UID: \"0771e950-74b9-45cb-a7be-ccb30c8c8c65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" Oct 30 02:30:01.508210 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.508069 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkvfw\" (UniqueName: \"kubernetes.io/projected/115cd7ff-80aa-4cd9-a247-9154fe7d117f-kube-api-access-vkvfw\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff\" (UID: \"115cd7ff-80aa-4cd9-a247-9154fe7d117f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" Oct 30 02:30:01.509375 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.509344 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clr67\" (UniqueName: \"kubernetes.io/projected/9e4909d0-f3ae-423a-a478-4ee5ce9734a1-kube-api-access-clr67\") pod \"maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc\" (UID: \"9e4909d0-f3ae-423a-a478-4ee5ce9734a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" Oct 30 02:30:01.511419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.510610 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq6kr\" (UniqueName: \"kubernetes.io/projected/512d72f9-4883-4d39-9df8-9282f56da2ac-kube-api-access-wq6kr\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr\" (UID: \"512d72f9-4883-4d39-9df8-9282f56da2ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" Oct 30 02:30:01.576711 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.576680 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" Oct 30 02:30:01.586995 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.586933 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" Oct 30 02:30:01.602528 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.599324 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4cvr\" (UniqueName: \"kubernetes.io/projected/96bc88e1-7b35-4b54-b12f-69227ae41a88-kube-api-access-v4cvr\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj\" (UID: \"96bc88e1-7b35-4b54-b12f-69227ae41a88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" Oct 30 02:30:01.602528 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.601875 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc"] Oct 30 02:30:01.609973 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.609946 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf"] Oct 30 02:30:01.689111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.689030 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" Oct 30 02:30:01.720213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.719998 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" Oct 30 02:30:01.738972 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.738802 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" Oct 30 02:30:01.752117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.751921 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" Oct 30 02:30:01.796532 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.796468 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw"] Oct 30 02:30:01.912545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.900930 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6"] Oct 30 02:30:01.955604 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:01.955567 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq"] Oct 30 02:30:02.323349 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:02.314174 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:02.938870 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:02.938687 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf"] Oct 30 02:30:02.938870 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:02.938754 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp"] Oct 30 02:30:02.953943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:02.953911 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" event={"ID":"512be8ad-51e7-4919-a456-9fcfafab46bd","Type":"ContainerStarted","Data":"be7b281637325fb5da2e13a7d3d544162ff987003df39f297f375018cbc9832e"} Oct 30 02:30:02.960603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:02.960192 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh"] Oct 30 02:30:02.994080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:02.994036 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" event={"ID":"e5882ea9-fe2e-4587-abb7-bb14939f53f6","Type":"ContainerStarted","Data":"3d2b005d212b83bbbef2d13ea9cab2f86525bf09a8d277c28ef63d9b6f7f39f2"} Oct 30 02:30:02.999401 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:02.999369 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" event={"ID":"01d4f72c-dc27-4988-b301-873c276855bd","Type":"ContainerStarted","Data":"c514c46fd577179a57c3681a35c5ef5aab6d148e58fae47ab874381283144a61"} Oct 30 02:30:03.094384 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:03.094351 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd"] Oct 30 02:30:03.129724 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:03.102172 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z"] Oct 30 02:30:03.462180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:03.461448 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn"] Oct 30 02:30:03.472687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:03.472646 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g"] Oct 30 02:30:03.476695 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:03.476665 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw"] Oct 30 02:30:03.554469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:03.554209 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn"] Oct 30 02:30:03.558708 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:03.558603 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc"] Oct 30 02:30:03.651058 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:03.651023 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr"] Oct 30 02:30:03.679166 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:30:03.679053 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512d72f9_4883_4d39_9df8_9282f56da2ac.slice/crio-55656aeb1553dcae558c0ccf770bd981151adc537fb4ff2a1a68935b03e1cddf WatchSource:0}: Error finding container 55656aeb1553dcae558c0ccf770bd981151adc537fb4ff2a1a68935b03e1cddf: Status 404 returned error can't find the container with id 55656aeb1553dcae558c0ccf770bd981151adc537fb4ff2a1a68935b03e1cddf Oct 30 02:30:03.737935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:03.732719 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt"] Oct 30 02:30:03.737935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:03.732814 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp"] Oct 30 02:30:03.740686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:03.740651 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k"] Oct 30 02:30:03.861688 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:03.861637 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc"] Oct 30 02:30:03.867269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:03.867240 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj"] Oct 30 02:30:03.872546 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:03.872519 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff"] Oct 30 02:30:03.949526 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:30:03.949486 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4909d0_f3ae_423a_a478_4ee5ce9734a1.slice/crio-6dfed4a59b4d7ed707d77f54a717104f003339736e547b12758b60d6801f72ad WatchSource:0}: Error finding container 6dfed4a59b4d7ed707d77f54a717104f003339736e547b12758b60d6801f72ad: Status 404 returned error can't find the container with id 6dfed4a59b4d7ed707d77f54a717104f003339736e547b12758b60d6801f72ad Oct 30 02:30:03.951085 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:03.951059 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs"] Oct 30 02:30:03.958601 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:03.958532 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2"] Oct 30 02:30:03.989423 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:03.989396 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf"] Oct 30 02:30:04.061848 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.061815 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:04.064887 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.064800 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" event={"ID":"046ebf9b-3d7e-4c67-bfcf-8545980b59ff","Type":"ContainerStarted","Data":"da8569ee276fbe190f7064e8444c12bb71d787d04c815d2dce451fb744fc932f"} Oct 30 02:30:04.068164 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.068133 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" event={"ID":"be309ab3-73d7-42c0-92a8-0fc686504795","Type":"ContainerStarted","Data":"97b78f0329c3df4253bb8d2403a4bdb3f979fc455367f30dd74d31454f7ecb82"} Oct 30 02:30:04.072042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.071054 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" event={"ID":"a1f42aec-2169-46f1-a837-039b920ac828","Type":"ContainerStarted","Data":"97d25e8634aa25fc3c5d6e705150e7d9cc637738840c20e207c6d9b7d18145f2"} Oct 30 02:30:04.072042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.071153 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" event={"ID":"a1f42aec-2169-46f1-a837-039b920ac828","Type":"ContainerStarted","Data":"c4e32547cc1aa242ae6cfd751a46f855ce64edd28f37a1c16448214085511d8e"} Oct 30 02:30:04.073798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.073769 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" event={"ID":"5bce6d68-142e-46ef-b408-89e44ca0fc10","Type":"ContainerStarted","Data":"ccafd81f214e1efcef9fb02f4d99efd01629930a1bfcfac3af939333eaa3b5c5"} Oct 30 02:30:04.081281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.081252 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" event={"ID":"96bc88e1-7b35-4b54-b12f-69227ae41a88","Type":"ContainerStarted","Data":"4e96c5d1588076e5dc8703de7d0a0c90023fc4a602129ea39a8e21abac9e146a"} Oct 30 02:30:04.082839 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.082783 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" event={"ID":"2f3ead60-88ca-42af-9e25-7822589058e4","Type":"ContainerStarted","Data":"226f6f99f1eabfe31ac2b59e1494cb518c0b315f67d29890719a3a3963ab6b0c"} Oct 30 02:30:04.084389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.084368 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc" event={"ID":"6df3039d-5102-404f-b58f-eae1e79feb36","Type":"ContainerStarted","Data":"ba4d93314e7e727725b05fc44ad5dcf5fece5703ac4cd47f155b71d2f7175a37"} Oct 30 02:30:04.086290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.086240 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" event={"ID":"3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb","Type":"ContainerStarted","Data":"47b5fd8c5fc05c932ac941f4eb4181978e34ccfe0a21634be509345be2ba6217"} Oct 30 02:30:04.087602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.087582 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" event={"ID":"512d72f9-4883-4d39-9df8-9282f56da2ac","Type":"ContainerStarted","Data":"55656aeb1553dcae558c0ccf770bd981151adc537fb4ff2a1a68935b03e1cddf"} Oct 30 02:30:04.091175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.091116 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" event={"ID":"9e4909d0-f3ae-423a-a478-4ee5ce9734a1","Type":"ContainerStarted","Data":"6dfed4a59b4d7ed707d77f54a717104f003339736e547b12758b60d6801f72ad"} Oct 30 02:30:04.091866 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:30:04.091841 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0771e950_74b9_45cb_a7be_ccb30c8c8c65.slice/crio-3bf9b239f0b8313a68f73a1ef111e1da1b76a1896fddb9abd828fdc4d6ea2d79 WatchSource:0}: Error finding container 3bf9b239f0b8313a68f73a1ef111e1da1b76a1896fddb9abd828fdc4d6ea2d79: Status 404 returned error can't find the container with id 3bf9b239f0b8313a68f73a1ef111e1da1b76a1896fddb9abd828fdc4d6ea2d79 Oct 30 02:30:04.129631 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.119988 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" podStartSLOduration=4.119968793 podStartE2EDuration="4.119968793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:04.117305829 +0000 UTC m=+3633.696096911" watchObservedRunningTime="2026-10-30 02:30:04.119968793 +0000 UTC m=+3633.698759875" Oct 30 02:30:04.255705 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.255026 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" event={"ID":"e5882ea9-fe2e-4587-abb7-bb14939f53f6","Type":"ContainerStarted","Data":"085be5d1f60316d1b76554f1dd2bb13ee5c6d0c644c8e2f2997e129f4f54213a"} Oct 30 02:30:04.261190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.258342 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" event={"ID":"daaa2b88-a49a-4a88-8325-33307e969074","Type":"ContainerStarted","Data":"6ae2cb2102a68c0bdd80c4b22a45b2edc3ed8ad5e88413a988280ed61a139d0f"} Oct 30 02:30:04.261190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.260551 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" event={"ID":"01d4f72c-dc27-4988-b301-873c276855bd","Type":"ContainerStarted","Data":"6ece01febe2804f449320c041a777cddcb32d569f8525c57c13f94420b247d6c"} Oct 30 02:30:04.261190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.261116 5998 scope.go:122] "RemoveContainer" containerID="6ece01febe2804f449320c041a777cddcb32d569f8525c57c13f94420b247d6c" Oct 30 02:30:04.294667 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.286254 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" podStartSLOduration=4.286237923 podStartE2EDuration="4.286237923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:04.228799847 +0000 UTC m=+3633.807590929" watchObservedRunningTime="2026-10-30 02:30:04.286237923 +0000 UTC m=+3633.865029000" Oct 30 02:30:04.336374 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.336308 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" podStartSLOduration=4.336283917 podStartE2EDuration="4.336283917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:04.299594695 +0000 UTC m=+3633.878385777" watchObservedRunningTime="2026-10-30 02:30:04.336283917 +0000 UTC m=+3633.915075000" Oct 30 02:30:04.363123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.348214 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:30:04.363123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.359826 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" event={"ID":"115cd7ff-80aa-4cd9-a247-9154fe7d117f","Type":"ContainerStarted","Data":"8e281291caa5ea8f17eea9b50df8e516bf91b7e76d023c3562512865aee1179f"} Oct 30 02:30:04.372494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.371398 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" event={"ID":"a105635a-7115-4c02-aa79-60516391579b","Type":"ContainerStarted","Data":"66d4cb23310319254484218b1151cad3fcc4027edd2df45bcda505fa8c67713d"} Oct 30 02:30:04.387154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.387119 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:30:04.397836 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.397748 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" event={"ID":"8e81c096-2704-4d9e-80b8-9b1f05e7151f","Type":"ContainerStarted","Data":"1e46e96864d9a3f64d99ff69936381ce4eba2dbe0a36b56a092a6e7e423b63b0"} Oct 30 02:30:04.469009 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.468796 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" event={"ID":"3f69dd9b-8eda-4368-b693-33c5b9721710","Type":"ContainerStarted","Data":"2105dbefb0bf07c84502095d1c58c9435244f3e82dbfeb979e7230a2a9403970"} Oct 30 02:30:04.472727 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.472406 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" event={"ID":"860ba768-1529-403e-ba81-13395d6cee83","Type":"ContainerStarted","Data":"8f9c04e22c707972485b92534f46d7bb0d4c00882459986ee66e3aa855bd7042"} Oct 30 02:30:04.474977 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.474944 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" event={"ID":"7ff76844-137e-44ec-b146-4d9c84997ada","Type":"ContainerStarted","Data":"05d3493c6ec50e7c22d00f83b3523aca4959bdbe0846e76207da0faed453f61c"} Oct 30 02:30:04.477506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.477471 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" event={"ID":"1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee","Type":"ContainerStarted","Data":"b0fe849dacf17e2b7ce2b4ea60f045619dbbcbbf5ddea7f0d204333d60c15532"} Oct 30 02:30:04.479790 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.479761 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" event={"ID":"c9b0ea41-0e61-456d-bbce-6ee7145552ea","Type":"ContainerStarted","Data":"d12e5e487d443806e4fd2e70fb6f93faf5a798386673f7171cbe6e5094afb2ad"} Oct 30 02:30:04.569441 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.569356 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" podStartSLOduration=4.569325992 podStartE2EDuration="4.569325992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:04.511878697 +0000 UTC m=+3634.090669779" watchObservedRunningTime="2026-10-30 02:30:04.569325992 +0000 UTC m=+3634.148117074" Oct 30 02:30:04.577321 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.577273 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" podStartSLOduration=4.577260265 podStartE2EDuration="4.577260265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:04.564357916 +0000 UTC m=+3634.143148996" watchObservedRunningTime="2026-10-30 02:30:04.577260265 +0000 UTC m=+3634.156051346" Oct 30 02:30:04.661052 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.660983 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" podStartSLOduration=4.660960334 podStartE2EDuration="4.660960334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:04.613118654 +0000 UTC m=+3634.191909736" watchObservedRunningTime="2026-10-30 02:30:04.660960334 +0000 UTC m=+3634.239751416" Oct 30 02:30:04.670849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:04.670796 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" podStartSLOduration=4.670775989 podStartE2EDuration="4.670775989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:04.667477778 +0000 UTC m=+3634.246268860" watchObservedRunningTime="2026-10-30 02:30:04.670775989 +0000 UTC m=+3634.249567071" Oct 30 02:30:05.359671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.359637 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:30:05.498710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.498674 5998 generic.go:356] "Generic (PLEG): container finished" podID="6df3039d-5102-404f-b58f-eae1e79feb36" containerID="368d869c3c9bad60d1b028dfa63f8a8ce628f21b43bc9f03fa7bf13c0027c001" exitCode=60 Oct 30 02:30:05.498878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.498754 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc" event={"ID":"6df3039d-5102-404f-b58f-eae1e79feb36","Type":"ContainerDied","Data":"368d869c3c9bad60d1b028dfa63f8a8ce628f21b43bc9f03fa7bf13c0027c001"} Oct 30 02:30:05.499577 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.499224 5998 scope.go:122] "RemoveContainer" containerID="368d869c3c9bad60d1b028dfa63f8a8ce628f21b43bc9f03fa7bf13c0027c001" Oct 30 02:30:05.575743 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.575702 5998 generic.go:356] "Generic (PLEG): container finished" podID="3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb" containerID="37be78c89daf9617b3abde34f17cc4fb07f593213bfbde659bd70b302b993cf1" exitCode=60 Oct 30 02:30:05.576256 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.575868 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" event={"ID":"3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb","Type":"ContainerDied","Data":"37be78c89daf9617b3abde34f17cc4fb07f593213bfbde659bd70b302b993cf1"} Oct 30 02:30:05.576903 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.576634 5998 scope.go:122] "RemoveContainer" containerID="37be78c89daf9617b3abde34f17cc4fb07f593213bfbde659bd70b302b993cf1" Oct 30 02:30:05.581687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.581652 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" event={"ID":"0771e950-74b9-45cb-a7be-ccb30c8c8c65","Type":"ContainerStarted","Data":"3bf9b239f0b8313a68f73a1ef111e1da1b76a1896fddb9abd828fdc4d6ea2d79"} Oct 30 02:30:05.589427 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.589396 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" event={"ID":"512d72f9-4883-4d39-9df8-9282f56da2ac","Type":"ContainerStarted","Data":"4d235597540bf96806bdc9dd108132d5f98a035aba4ffabadbd717d9d1119a71"} Oct 30 02:30:05.611360 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.604715 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" event={"ID":"9e4909d0-f3ae-423a-a478-4ee5ce9734a1","Type":"ContainerStarted","Data":"0a89aced9636173d35bda68cce1b6bb74220865730c332ddc9db79422bc26189"} Oct 30 02:30:05.665797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.665762 5998 generic.go:356] "Generic (PLEG): container finished" podID="e5882ea9-fe2e-4587-abb7-bb14939f53f6" containerID="085be5d1f60316d1b76554f1dd2bb13ee5c6d0c644c8e2f2997e129f4f54213a" exitCode=60 Oct 30 02:30:05.666063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.666046 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" event={"ID":"e5882ea9-fe2e-4587-abb7-bb14939f53f6","Type":"ContainerDied","Data":"085be5d1f60316d1b76554f1dd2bb13ee5c6d0c644c8e2f2997e129f4f54213a"} Oct 30 02:30:05.666709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.666690 5998 scope.go:122] "RemoveContainer" containerID="085be5d1f60316d1b76554f1dd2bb13ee5c6d0c644c8e2f2997e129f4f54213a" Oct 30 02:30:05.677281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.677218 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" podStartSLOduration=5.6771983200000005 podStartE2EDuration="5.67719832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:05.66748394 +0000 UTC m=+3635.246275021" watchObservedRunningTime="2026-10-30 02:30:05.67719832 +0000 UTC m=+3635.255989406" Oct 30 02:30:05.683847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.683812 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" event={"ID":"daaa2b88-a49a-4a88-8325-33307e969074","Type":"ContainerStarted","Data":"22c3a3fbb220d8eca90cab8b3bdff426bfec46a1e2292cebb40368ffff31aa83"} Oct 30 02:30:05.687857 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.687829 5998 generic.go:356] "Generic (PLEG): container finished" podID="01d4f72c-dc27-4988-b301-873c276855bd" containerID="6ece01febe2804f449320c041a777cddcb32d569f8525c57c13f94420b247d6c" exitCode=60 Oct 30 02:30:05.687969 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.687925 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" event={"ID":"01d4f72c-dc27-4988-b301-873c276855bd","Type":"ContainerDied","Data":"6ece01febe2804f449320c041a777cddcb32d569f8525c57c13f94420b247d6c"} Oct 30 02:30:05.690050 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.690028 5998 generic.go:356] "Generic (PLEG): container finished" podID="a105635a-7115-4c02-aa79-60516391579b" containerID="7dff198935494a5c60e7d09ae675624a72d96442b5576e39cf49744a6a724b23" exitCode=60 Oct 30 02:30:05.690445 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.690423 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" event={"ID":"a105635a-7115-4c02-aa79-60516391579b","Type":"ContainerDied","Data":"7dff198935494a5c60e7d09ae675624a72d96442b5576e39cf49744a6a724b23"} Oct 30 02:30:05.690820 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.690807 5998 scope.go:122] "RemoveContainer" containerID="7dff198935494a5c60e7d09ae675624a72d96442b5576e39cf49744a6a724b23" Oct 30 02:30:05.694586 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.694560 5998 generic.go:356] "Generic (PLEG): container finished" podID="8e81c096-2704-4d9e-80b8-9b1f05e7151f" containerID="7ddbdd413c7ac1dcf9ccafb8874e40a89701e3c518eff89900fec74caf6b3455" exitCode=60 Oct 30 02:30:05.694824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.694804 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" event={"ID":"8e81c096-2704-4d9e-80b8-9b1f05e7151f","Type":"ContainerDied","Data":"7ddbdd413c7ac1dcf9ccafb8874e40a89701e3c518eff89900fec74caf6b3455"} Oct 30 02:30:05.696608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.696581 5998 scope.go:122] "RemoveContainer" containerID="7ddbdd413c7ac1dcf9ccafb8874e40a89701e3c518eff89900fec74caf6b3455" Oct 30 02:30:05.742543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.724334 5998 generic.go:356] "Generic (PLEG): container finished" podID="3f69dd9b-8eda-4368-b693-33c5b9721710" containerID="6fc4f0974222bd2edb4a418038e65eb9d9173a437a01710f4b91ef2c2d89333d" exitCode=60 Oct 30 02:30:05.742543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.724437 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" event={"ID":"3f69dd9b-8eda-4368-b693-33c5b9721710","Type":"ContainerDied","Data":"6fc4f0974222bd2edb4a418038e65eb9d9173a437a01710f4b91ef2c2d89333d"} Oct 30 02:30:05.742543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.724970 5998 scope.go:122] "RemoveContainer" containerID="6fc4f0974222bd2edb4a418038e65eb9d9173a437a01710f4b91ef2c2d89333d" Oct 30 02:30:05.742543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.729020 5998 generic.go:356] "Generic (PLEG): container finished" podID="860ba768-1529-403e-ba81-13395d6cee83" containerID="4284dcc22fd4e42eebddee7d5ceb78fa68fa40b5c0a458646f32bc27ef538eee" exitCode=60 Oct 30 02:30:05.742543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.729555 5998 scope.go:122] "RemoveContainer" containerID="4284dcc22fd4e42eebddee7d5ceb78fa68fa40b5c0a458646f32bc27ef538eee" Oct 30 02:30:05.742543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.730080 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" event={"ID":"860ba768-1529-403e-ba81-13395d6cee83","Type":"ContainerDied","Data":"4284dcc22fd4e42eebddee7d5ceb78fa68fa40b5c0a458646f32bc27ef538eee"} Oct 30 02:30:05.742543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.732286 5998 generic.go:356] "Generic (PLEG): container finished" podID="7ff76844-137e-44ec-b146-4d9c84997ada" containerID="b961458c52fddbdde3196ec9997ec8324461dab8a29006000adaf76fddf80ee2" exitCode=60 Oct 30 02:30:05.742543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.732404 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" event={"ID":"7ff76844-137e-44ec-b146-4d9c84997ada","Type":"ContainerDied","Data":"b961458c52fddbdde3196ec9997ec8324461dab8a29006000adaf76fddf80ee2"} Oct 30 02:30:05.742543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.732869 5998 scope.go:122] "RemoveContainer" containerID="b961458c52fddbdde3196ec9997ec8324461dab8a29006000adaf76fddf80ee2" Oct 30 02:30:05.742543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.736571 5998 generic.go:356] "Generic (PLEG): container finished" podID="1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee" containerID="8b52af96be2391667b175cec58d51d6207ed530c0928bc72e6db21c3505181a1" exitCode=60 Oct 30 02:30:05.742543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.736645 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" event={"ID":"1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee","Type":"ContainerDied","Data":"8b52af96be2391667b175cec58d51d6207ed530c0928bc72e6db21c3505181a1"} Oct 30 02:30:05.742543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.737025 5998 scope.go:122] "RemoveContainer" containerID="8b52af96be2391667b175cec58d51d6207ed530c0928bc72e6db21c3505181a1" Oct 30 02:30:05.742543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.739673 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" event={"ID":"c9b0ea41-0e61-456d-bbce-6ee7145552ea","Type":"ContainerStarted","Data":"dd1f75e17ef793719a2a64626fbd646480e9a7f8f29bf8b13a785bb78b9fbdc6"} Oct 30 02:30:05.761882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.761811 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" podStartSLOduration=5.761792293 podStartE2EDuration="5.761792293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:05.720335226 +0000 UTC m=+3635.299126307" watchObservedRunningTime="2026-10-30 02:30:05.761792293 +0000 UTC m=+3635.340583374" Oct 30 02:30:05.770032 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.769983 5998 generic.go:356] "Generic (PLEG): container finished" podID="046ebf9b-3d7e-4c67-bfcf-8545980b59ff" containerID="86743c3c7f633eaaa1a175d40d1c3cdaa4dc80bf86dc193a59fe5fd8ca94f969" exitCode=60 Oct 30 02:30:05.770261 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.770115 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" event={"ID":"046ebf9b-3d7e-4c67-bfcf-8545980b59ff","Type":"ContainerDied","Data":"86743c3c7f633eaaa1a175d40d1c3cdaa4dc80bf86dc193a59fe5fd8ca94f969"} Oct 30 02:30:05.770645 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.770615 5998 scope.go:122] "RemoveContainer" containerID="86743c3c7f633eaaa1a175d40d1c3cdaa4dc80bf86dc193a59fe5fd8ca94f969" Oct 30 02:30:05.772719 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.772691 5998 generic.go:356] "Generic (PLEG): container finished" podID="be309ab3-73d7-42c0-92a8-0fc686504795" containerID="6a5cdd480dcc873dd356404530fc4d69071654947dffac2764e1d31ca5c7e149" exitCode=60 Oct 30 02:30:05.773017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.772892 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" event={"ID":"be309ab3-73d7-42c0-92a8-0fc686504795","Type":"ContainerDied","Data":"6a5cdd480dcc873dd356404530fc4d69071654947dffac2764e1d31ca5c7e149"} Oct 30 02:30:05.773289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.773255 5998 scope.go:122] "RemoveContainer" containerID="6a5cdd480dcc873dd356404530fc4d69071654947dffac2764e1d31ca5c7e149" Oct 30 02:30:05.775303 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.775281 5998 generic.go:356] "Generic (PLEG): container finished" podID="a1f42aec-2169-46f1-a837-039b920ac828" containerID="97d25e8634aa25fc3c5d6e705150e7d9cc637738840c20e207c6d9b7d18145f2" exitCode=60 Oct 30 02:30:05.775943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.775916 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" event={"ID":"a1f42aec-2169-46f1-a837-039b920ac828","Type":"ContainerDied","Data":"97d25e8634aa25fc3c5d6e705150e7d9cc637738840c20e207c6d9b7d18145f2"} Oct 30 02:30:05.776037 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.775747 5998 scope.go:122] "RemoveContainer" containerID="97d25e8634aa25fc3c5d6e705150e7d9cc637738840c20e207c6d9b7d18145f2" Oct 30 02:30:05.777678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.777654 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:05.779166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.779144 5998 generic.go:356] "Generic (PLEG): container finished" podID="512be8ad-51e7-4919-a456-9fcfafab46bd" containerID="431bfb2cacf1d8f5eca603577dd4019e6a593c3e4cc2fbb1f61bbdd5aceccb41" exitCode=60 Oct 30 02:30:05.779402 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.779374 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" event={"ID":"512be8ad-51e7-4919-a456-9fcfafab46bd","Type":"ContainerDied","Data":"431bfb2cacf1d8f5eca603577dd4019e6a593c3e4cc2fbb1f61bbdd5aceccb41"} Oct 30 02:30:05.779864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.779845 5998 scope.go:122] "RemoveContainer" containerID="431bfb2cacf1d8f5eca603577dd4019e6a593c3e4cc2fbb1f61bbdd5aceccb41" Oct 30 02:30:05.950452 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:05.950394 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" podStartSLOduration=5.903323129 podStartE2EDuration="5.903323129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:05.891180055 +0000 UTC m=+3635.469971137" watchObservedRunningTime="2026-10-30 02:30:05.903323129 +0000 UTC m=+3635.482114211" Oct 30 02:30:06.361579 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.361507 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" podStartSLOduration=6.361484665 podStartE2EDuration="6.361484665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:06.351126474 +0000 UTC m=+3635.929917555" watchObservedRunningTime="2026-10-30 02:30:06.361484665 +0000 UTC m=+3635.940275748" Oct 30 02:30:06.792749 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.792705 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" event={"ID":"0771e950-74b9-45cb-a7be-ccb30c8c8c65","Type":"ContainerStarted","Data":"da39fef5cb879a319437b3c2f9cede8e39e485c9f0ede3d651eba984fc314a5a"} Oct 30 02:30:06.794710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.794681 5998 generic.go:356] "Generic (PLEG): container finished" podID="512d72f9-4883-4d39-9df8-9282f56da2ac" containerID="4d235597540bf96806bdc9dd108132d5f98a035aba4ffabadbd717d9d1119a71" exitCode=60 Oct 30 02:30:06.794847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.794759 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" event={"ID":"512d72f9-4883-4d39-9df8-9282f56da2ac","Type":"ContainerDied","Data":"4d235597540bf96806bdc9dd108132d5f98a035aba4ffabadbd717d9d1119a71"} Oct 30 02:30:06.795163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.795141 5998 scope.go:122] "RemoveContainer" containerID="4d235597540bf96806bdc9dd108132d5f98a035aba4ffabadbd717d9d1119a71" Oct 30 02:30:06.796738 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.796707 5998 generic.go:356] "Generic (PLEG): container finished" podID="9e4909d0-f3ae-423a-a478-4ee5ce9734a1" containerID="0a89aced9636173d35bda68cce1b6bb74220865730c332ddc9db79422bc26189" exitCode=60 Oct 30 02:30:06.796864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.796798 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" event={"ID":"9e4909d0-f3ae-423a-a478-4ee5ce9734a1","Type":"ContainerDied","Data":"0a89aced9636173d35bda68cce1b6bb74220865730c332ddc9db79422bc26189"} Oct 30 02:30:06.797082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.797062 5998 scope.go:122] "RemoveContainer" containerID="0a89aced9636173d35bda68cce1b6bb74220865730c332ddc9db79422bc26189" Oct 30 02:30:06.799217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.799194 5998 generic.go:356] "Generic (PLEG): container finished" podID="daaa2b88-a49a-4a88-8325-33307e969074" containerID="22c3a3fbb220d8eca90cab8b3bdff426bfec46a1e2292cebb40368ffff31aa83" exitCode=60 Oct 30 02:30:06.799320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.799283 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" event={"ID":"daaa2b88-a49a-4a88-8325-33307e969074","Type":"ContainerDied","Data":"22c3a3fbb220d8eca90cab8b3bdff426bfec46a1e2292cebb40368ffff31aa83"} Oct 30 02:30:06.799680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.799661 5998 scope.go:122] "RemoveContainer" containerID="22c3a3fbb220d8eca90cab8b3bdff426bfec46a1e2292cebb40368ffff31aa83" Oct 30 02:30:06.813765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.809759 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" event={"ID":"01d4f72c-dc27-4988-b301-873c276855bd","Type":"ContainerStarted","Data":"26c7ffac65a2098d82f0c88a36285c9676a5074e2def1c037e1b04c0bba86ef0"} Oct 30 02:30:06.813765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.811170 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:30:06.820677 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.820594 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" event={"ID":"115cd7ff-80aa-4cd9-a247-9154fe7d117f","Type":"ContainerStarted","Data":"c93f61213e974218a97064a9061f11f0869a78e8227079a6743ea373859cd085"} Oct 30 02:30:06.824789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.824759 5998 generic.go:356] "Generic (PLEG): container finished" podID="c9b0ea41-0e61-456d-bbce-6ee7145552ea" containerID="dd1f75e17ef793719a2a64626fbd646480e9a7f8f29bf8b13a785bb78b9fbdc6" exitCode=60 Oct 30 02:30:06.825101 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.825063 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" event={"ID":"c9b0ea41-0e61-456d-bbce-6ee7145552ea","Type":"ContainerDied","Data":"dd1f75e17ef793719a2a64626fbd646480e9a7f8f29bf8b13a785bb78b9fbdc6"} Oct 30 02:30:06.825698 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.825678 5998 scope.go:122] "RemoveContainer" containerID="dd1f75e17ef793719a2a64626fbd646480e9a7f8f29bf8b13a785bb78b9fbdc6" Oct 30 02:30:06.830996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.830130 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" event={"ID":"5bce6d68-142e-46ef-b408-89e44ca0fc10","Type":"ContainerStarted","Data":"4899e73bb09c44c832db82e0c44fcd76aa34557cb93cc9f03819fc52e6a483e2"} Oct 30 02:30:06.836779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.833435 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" event={"ID":"96bc88e1-7b35-4b54-b12f-69227ae41a88","Type":"ContainerStarted","Data":"c8ad237c55c76c7ef258fc7442b4cf713f2d11f4b65cabb9b27679a9d8164a4e"} Oct 30 02:30:06.850116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.840801 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" event={"ID":"2f3ead60-88ca-42af-9e25-7822589058e4","Type":"ContainerStarted","Data":"73ec7df62572677493e234e0dc20d5a86b06f7ec9019e85d14a51f436d0f2425"} Oct 30 02:30:06.856798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.856738 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" podStartSLOduration=6.856720341 podStartE2EDuration="6.856720341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:06.836386338 +0000 UTC m=+3636.415177420" watchObservedRunningTime="2026-10-30 02:30:06.856720341 +0000 UTC m=+3636.435511421" Oct 30 02:30:06.875504 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.875481 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:30:06.941815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.941742 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" podStartSLOduration=6.941721265 podStartE2EDuration="6.941721265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:06.878416681 +0000 UTC m=+3636.457207762" watchObservedRunningTime="2026-10-30 02:30:06.941721265 +0000 UTC m=+3636.520512350" Oct 30 02:30:06.954394 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:06.954321 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" podStartSLOduration=6.954302298 podStartE2EDuration="6.954302298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:06.940857767 +0000 UTC m=+3636.519648849" watchObservedRunningTime="2026-10-30 02:30:06.954302298 +0000 UTC m=+3636.533093380" Oct 30 02:30:07.083367 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.083263 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" podStartSLOduration=7.083248112 podStartE2EDuration="7.083248112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:07.077875141 +0000 UTC m=+3636.656666224" watchObservedRunningTime="2026-10-30 02:30:07.083248112 +0000 UTC m=+3636.662039192" Oct 30 02:30:07.244213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.244135 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" podStartSLOduration=7.244114913 podStartE2EDuration="7.244114913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:07.235398515 +0000 UTC m=+3636.814189597" watchObservedRunningTime="2026-10-30 02:30:07.244114913 +0000 UTC m=+3636.822906000" Oct 30 02:30:07.343523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.340970 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" podStartSLOduration=7.340949513 podStartE2EDuration="7.340949513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:07.322539005 +0000 UTC m=+3636.901330087" watchObservedRunningTime="2026-10-30 02:30:07.340949513 +0000 UTC m=+3636.919740595" Oct 30 02:30:07.488361 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.488320 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:07.861663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.861629 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:30:07.866061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.866023 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" event={"ID":"e5882ea9-fe2e-4587-abb7-bb14939f53f6","Type":"ContainerStarted","Data":"1e12b2bcf54976d519b107ab2a34702c508caeb6f5ebc61b072d92a53e171474"} Oct 30 02:30:07.868312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.868239 5998 generic.go:356] "Generic (PLEG): container finished" podID="01d4f72c-dc27-4988-b301-873c276855bd" containerID="26c7ffac65a2098d82f0c88a36285c9676a5074e2def1c037e1b04c0bba86ef0" exitCode=60 Oct 30 02:30:07.868312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.868275 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" event={"ID":"01d4f72c-dc27-4988-b301-873c276855bd","Type":"ContainerDied","Data":"26c7ffac65a2098d82f0c88a36285c9676a5074e2def1c037e1b04c0bba86ef0"} Oct 30 02:30:07.868479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.868340 5998 scope.go:122] "RemoveContainer" containerID="6ece01febe2804f449320c041a777cddcb32d569f8525c57c13f94420b247d6c" Oct 30 02:30:07.868616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.868590 5998 scope.go:122] "RemoveContainer" containerID="26c7ffac65a2098d82f0c88a36285c9676a5074e2def1c037e1b04c0bba86ef0" Oct 30 02:30:07.868965 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:07.868934 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw_odh-ai-gateway-infra(01d4f72c-dc27-4988-b301-873c276855bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" podUID="01d4f72c-dc27-4988-b301-873c276855bd" Oct 30 02:30:07.870382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.870352 5998 generic.go:356] "Generic (PLEG): container finished" podID="115cd7ff-80aa-4cd9-a247-9154fe7d117f" containerID="c93f61213e974218a97064a9061f11f0869a78e8227079a6743ea373859cd085" exitCode=60 Oct 30 02:30:07.870484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.870446 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" event={"ID":"115cd7ff-80aa-4cd9-a247-9154fe7d117f","Type":"ContainerDied","Data":"c93f61213e974218a97064a9061f11f0869a78e8227079a6743ea373859cd085"} Oct 30 02:30:07.870753 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.870733 5998 scope.go:122] "RemoveContainer" containerID="c93f61213e974218a97064a9061f11f0869a78e8227079a6743ea373859cd085" Oct 30 02:30:07.875300 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.875265 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" event={"ID":"a105635a-7115-4c02-aa79-60516391579b","Type":"ContainerStarted","Data":"bfd51dfd12fc0d920e83de5d303f6a3ee47c919afd18e5c105fb8e82a1d9fd9d"} Oct 30 02:30:07.877486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.877462 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" event={"ID":"8e81c096-2704-4d9e-80b8-9b1f05e7151f","Type":"ContainerStarted","Data":"9b3267105635039f6fce14887cdffa568d8ed5d82c9b4b35612a9f9fc490dbdb"} Oct 30 02:30:07.880003 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.879976 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" event={"ID":"860ba768-1529-403e-ba81-13395d6cee83","Type":"ContainerStarted","Data":"1e8cc9dcedae3a0189b91de81ab12c7c76a38fa54f0f541997685c79fda78c03"} Oct 30 02:30:07.888055 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.888024 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" event={"ID":"7ff76844-137e-44ec-b146-4d9c84997ada","Type":"ContainerStarted","Data":"c2870e68e7edfdb62185f69598aea4df8767568a247e024b00e20d7dfc7c02ed"} Oct 30 02:30:07.891711 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.891683 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" event={"ID":"1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee","Type":"ContainerStarted","Data":"a9307cc5ccab074ce26367e32ecd77e28e9306b249bb36ed62f49cdb022a85db"} Oct 30 02:30:07.893878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.893855 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" event={"ID":"be309ab3-73d7-42c0-92a8-0fc686504795","Type":"ContainerStarted","Data":"e8c11b0e22deafb51a20cb90ebd0266e9fe0759616e63a29e7ec5eb0176f6462"} Oct 30 02:30:07.896131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.896067 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" event={"ID":"a1f42aec-2169-46f1-a837-039b920ac828","Type":"ContainerStarted","Data":"ef3e023f57ad7986d43f1756955746298bd01f8d4ec96b43589aa582cdb699b1"} Oct 30 02:30:07.898322 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.898295 5998 generic.go:356] "Generic (PLEG): container finished" podID="5bce6d68-142e-46ef-b408-89e44ca0fc10" containerID="4899e73bb09c44c832db82e0c44fcd76aa34557cb93cc9f03819fc52e6a483e2" exitCode=60 Oct 30 02:30:07.898723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.898703 5998 scope.go:122] "RemoveContainer" containerID="4899e73bb09c44c832db82e0c44fcd76aa34557cb93cc9f03819fc52e6a483e2" Oct 30 02:30:07.899185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.899164 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" event={"ID":"5bce6d68-142e-46ef-b408-89e44ca0fc10","Type":"ContainerDied","Data":"4899e73bb09c44c832db82e0c44fcd76aa34557cb93cc9f03819fc52e6a483e2"} Oct 30 02:30:07.970764 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.970728 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" event={"ID":"512be8ad-51e7-4919-a456-9fcfafab46bd","Type":"ContainerStarted","Data":"27c4826c0628cf90c7f034eec6af68694bbe65e42077d4f46bf99fa840712636"} Oct 30 02:30:07.976903 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.976746 5998 generic.go:356] "Generic (PLEG): container finished" podID="96bc88e1-7b35-4b54-b12f-69227ae41a88" containerID="c8ad237c55c76c7ef258fc7442b4cf713f2d11f4b65cabb9b27679a9d8164a4e" exitCode=60 Oct 30 02:30:07.976903 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.976842 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" event={"ID":"96bc88e1-7b35-4b54-b12f-69227ae41a88","Type":"ContainerDied","Data":"c8ad237c55c76c7ef258fc7442b4cf713f2d11f4b65cabb9b27679a9d8164a4e"} Oct 30 02:30:07.977614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.977226 5998 scope.go:122] "RemoveContainer" containerID="c8ad237c55c76c7ef258fc7442b4cf713f2d11f4b65cabb9b27679a9d8164a4e" Oct 30 02:30:07.982684 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.982636 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" podStartSLOduration=7.982619497 podStartE2EDuration="7.982619497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:07.973038357 +0000 UTC m=+3637.551829438" watchObservedRunningTime="2026-10-30 02:30:07.982619497 +0000 UTC m=+3637.561410581" Oct 30 02:30:07.989561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.987899 5998 generic.go:356] "Generic (PLEG): container finished" podID="2f3ead60-88ca-42af-9e25-7822589058e4" containerID="73ec7df62572677493e234e0dc20d5a86b06f7ec9019e85d14a51f436d0f2425" exitCode=60 Oct 30 02:30:07.989561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.987985 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" event={"ID":"2f3ead60-88ca-42af-9e25-7822589058e4","Type":"ContainerDied","Data":"73ec7df62572677493e234e0dc20d5a86b06f7ec9019e85d14a51f436d0f2425"} Oct 30 02:30:07.989561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.988565 5998 scope.go:122] "RemoveContainer" containerID="73ec7df62572677493e234e0dc20d5a86b06f7ec9019e85d14a51f436d0f2425" Oct 30 02:30:07.992063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.991877 5998 generic.go:356] "Generic (PLEG): container finished" podID="6df3039d-5102-404f-b58f-eae1e79feb36" containerID="1555edc8ec84f776cd24750e285b0d3a85649a39e915c06462a76670d55d26e0" exitCode=60 Oct 30 02:30:07.992063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.991994 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc" event={"ID":"6df3039d-5102-404f-b58f-eae1e79feb36","Type":"ContainerDied","Data":"1555edc8ec84f776cd24750e285b0d3a85649a39e915c06462a76670d55d26e0"} Oct 30 02:30:07.993617 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:07.992312 5998 scope.go:122] "RemoveContainer" containerID="1555edc8ec84f776cd24750e285b0d3a85649a39e915c06462a76670d55d26e0" Oct 30 02:30:07.993617 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:07.992710 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc_odh-ai-gateway-infra(6df3039d-5102-404f-b58f-eae1e79feb36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc" podUID="6df3039d-5102-404f-b58f-eae1e79feb36" Oct 30 02:30:08.069001 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:08.068930 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" event={"ID":"3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb","Type":"ContainerStarted","Data":"8ef843f3d22e1c1029643acdb1e32dfea707800156c7a4cd6d1b284c6c5b5637"} Oct 30 02:30:08.075117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:08.074411 5998 generic.go:356] "Generic (PLEG): container finished" podID="0771e950-74b9-45cb-a7be-ccb30c8c8c65" containerID="da39fef5cb879a319437b3c2f9cede8e39e485c9f0ede3d651eba984fc314a5a" exitCode=60 Oct 30 02:30:08.075505 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:08.075477 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" event={"ID":"0771e950-74b9-45cb-a7be-ccb30c8c8c65","Type":"ContainerDied","Data":"da39fef5cb879a319437b3c2f9cede8e39e485c9f0ede3d651eba984fc314a5a"} Oct 30 02:30:08.076456 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:08.075609 5998 scope.go:122] "RemoveContainer" containerID="da39fef5cb879a319437b3c2f9cede8e39e485c9f0ede3d651eba984fc314a5a" Oct 30 02:30:08.156013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:08.155932 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" podStartSLOduration=8.155889281 podStartE2EDuration="8.155889281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:08.044459225 +0000 UTC m=+3637.623250306" watchObservedRunningTime="2026-10-30 02:30:08.155889281 +0000 UTC m=+3637.734680368" Oct 30 02:30:08.528116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:08.527117 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" podStartSLOduration=8.527078262 podStartE2EDuration="8.527078262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:08.511416501 +0000 UTC m=+3638.090207584" watchObservedRunningTime="2026-10-30 02:30:08.527078262 +0000 UTC m=+3638.105869343" Oct 30 02:30:08.562864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:08.558799 5998 scope.go:122] "RemoveContainer" containerID="368d869c3c9bad60d1b028dfa63f8a8ce628f21b43bc9f03fa7bf13c0027c001" Oct 30 02:30:08.592591 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:08.592540 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" podStartSLOduration=8.592523824 podStartE2EDuration="8.592523824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:08.588998078 +0000 UTC m=+3638.167789159" watchObservedRunningTime="2026-10-30 02:30:08.592523824 +0000 UTC m=+3638.171314906" Oct 30 02:30:09.087604 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.087563 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" event={"ID":"046ebf9b-3d7e-4c67-bfcf-8545980b59ff","Type":"ContainerStarted","Data":"06ba2ce69cb9eb1821fef5fdb771d2512a2d3740b31562f876c764ccc8cec9bc"} Oct 30 02:30:09.089723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.089691 5998 generic.go:356] "Generic (PLEG): container finished" podID="be309ab3-73d7-42c0-92a8-0fc686504795" containerID="e8c11b0e22deafb51a20cb90ebd0266e9fe0759616e63a29e7ec5eb0176f6462" exitCode=60 Oct 30 02:30:09.089858 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.089766 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" event={"ID":"be309ab3-73d7-42c0-92a8-0fc686504795","Type":"ContainerDied","Data":"e8c11b0e22deafb51a20cb90ebd0266e9fe0759616e63a29e7ec5eb0176f6462"} Oct 30 02:30:09.089858 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.089821 5998 scope.go:122] "RemoveContainer" containerID="6a5cdd480dcc873dd356404530fc4d69071654947dffac2764e1d31ca5c7e149" Oct 30 02:30:09.090202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.090182 5998 scope.go:122] "RemoveContainer" containerID="e8c11b0e22deafb51a20cb90ebd0266e9fe0759616e63a29e7ec5eb0176f6462" Oct 30 02:30:09.090611 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:09.090589 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf_odh-ai-gateway-infra(be309ab3-73d7-42c0-92a8-0fc686504795)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" podUID="be309ab3-73d7-42c0-92a8-0fc686504795" Oct 30 02:30:09.092169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.092145 5998 generic.go:356] "Generic (PLEG): container finished" podID="a1f42aec-2169-46f1-a837-039b920ac828" containerID="ef3e023f57ad7986d43f1756955746298bd01f8d4ec96b43589aa582cdb699b1" exitCode=60 Oct 30 02:30:09.092263 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.092225 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" event={"ID":"a1f42aec-2169-46f1-a837-039b920ac828","Type":"ContainerDied","Data":"ef3e023f57ad7986d43f1756955746298bd01f8d4ec96b43589aa582cdb699b1"} Oct 30 02:30:09.092517 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.092502 5998 scope.go:122] "RemoveContainer" containerID="ef3e023f57ad7986d43f1756955746298bd01f8d4ec96b43589aa582cdb699b1" Oct 30 02:30:09.092914 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:09.092891 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd_odh-ai-gateway-infra(a1f42aec-2169-46f1-a837-039b920ac828)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" podUID="a1f42aec-2169-46f1-a837-039b920ac828" Oct 30 02:30:09.094155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.094135 5998 generic.go:356] "Generic (PLEG): container finished" podID="512be8ad-51e7-4919-a456-9fcfafab46bd" containerID="27c4826c0628cf90c7f034eec6af68694bbe65e42077d4f46bf99fa840712636" exitCode=60 Oct 30 02:30:09.094239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.094202 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" event={"ID":"512be8ad-51e7-4919-a456-9fcfafab46bd","Type":"ContainerDied","Data":"27c4826c0628cf90c7f034eec6af68694bbe65e42077d4f46bf99fa840712636"} Oct 30 02:30:09.094566 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.094547 5998 scope.go:122] "RemoveContainer" containerID="27c4826c0628cf90c7f034eec6af68694bbe65e42077d4f46bf99fa840712636" Oct 30 02:30:09.094820 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:09.094798 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq_odh-ai-gateway-infra(512be8ad-51e7-4919-a456-9fcfafab46bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" podUID="512be8ad-51e7-4919-a456-9fcfafab46bd" Oct 30 02:30:09.096361 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.096339 5998 scope.go:122] "RemoveContainer" containerID="1555edc8ec84f776cd24750e285b0d3a85649a39e915c06462a76670d55d26e0" Oct 30 02:30:09.096700 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:09.096674 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc_odh-ai-gateway-infra(6df3039d-5102-404f-b58f-eae1e79feb36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc" podUID="6df3039d-5102-404f-b58f-eae1e79feb36" Oct 30 02:30:09.098117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.098073 5998 generic.go:356] "Generic (PLEG): container finished" podID="3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb" containerID="8ef843f3d22e1c1029643acdb1e32dfea707800156c7a4cd6d1b284c6c5b5637" exitCode=60 Oct 30 02:30:09.098204 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.098123 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" event={"ID":"3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb","Type":"ContainerDied","Data":"8ef843f3d22e1c1029643acdb1e32dfea707800156c7a4cd6d1b284c6c5b5637"} Oct 30 02:30:09.098387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.098370 5998 scope.go:122] "RemoveContainer" containerID="8ef843f3d22e1c1029643acdb1e32dfea707800156c7a4cd6d1b284c6c5b5637" Oct 30 02:30:09.098647 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:09.098622 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn_odh-ai-gateway-infra(3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" podUID="3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb" Oct 30 02:30:09.102239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.100365 5998 generic.go:356] "Generic (PLEG): container finished" podID="e5882ea9-fe2e-4587-abb7-bb14939f53f6" containerID="1e12b2bcf54976d519b107ab2a34702c508caeb6f5ebc61b072d92a53e171474" exitCode=60 Oct 30 02:30:09.102239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.100510 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" event={"ID":"e5882ea9-fe2e-4587-abb7-bb14939f53f6","Type":"ContainerDied","Data":"1e12b2bcf54976d519b107ab2a34702c508caeb6f5ebc61b072d92a53e171474"} Oct 30 02:30:09.102239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.100796 5998 scope.go:122] "RemoveContainer" containerID="1e12b2bcf54976d519b107ab2a34702c508caeb6f5ebc61b072d92a53e171474" Oct 30 02:30:09.102239 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:09.100987 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6_odh-ai-gateway-infra(e5882ea9-fe2e-4587-abb7-bb14939f53f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" podUID="e5882ea9-fe2e-4587-abb7-bb14939f53f6" Oct 30 02:30:09.104358 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.104334 5998 scope.go:122] "RemoveContainer" containerID="26c7ffac65a2098d82f0c88a36285c9676a5074e2def1c037e1b04c0bba86ef0" Oct 30 02:30:09.104652 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:09.104622 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw_odh-ai-gateway-infra(01d4f72c-dc27-4988-b301-873c276855bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" podUID="01d4f72c-dc27-4988-b301-873c276855bd" Oct 30 02:30:09.107763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.107712 5998 generic.go:356] "Generic (PLEG): container finished" podID="a105635a-7115-4c02-aa79-60516391579b" containerID="bfd51dfd12fc0d920e83de5d303f6a3ee47c919afd18e5c105fb8e82a1d9fd9d" exitCode=60 Oct 30 02:30:09.110974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.108021 5998 scope.go:122] "RemoveContainer" containerID="bfd51dfd12fc0d920e83de5d303f6a3ee47c919afd18e5c105fb8e82a1d9fd9d" Oct 30 02:30:09.110974 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:09.108393 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt_odh-ai-gateway-infra(a105635a-7115-4c02-aa79-60516391579b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" podUID="a105635a-7115-4c02-aa79-60516391579b" Oct 30 02:30:09.110974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.108498 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" event={"ID":"a105635a-7115-4c02-aa79-60516391579b","Type":"ContainerDied","Data":"bfd51dfd12fc0d920e83de5d303f6a3ee47c919afd18e5c105fb8e82a1d9fd9d"} Oct 30 02:30:09.115326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.115241 5998 scope.go:122] "RemoveContainer" containerID="97d25e8634aa25fc3c5d6e705150e7d9cc637738840c20e207c6d9b7d18145f2" Oct 30 02:30:09.120268 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.120052 5998 generic.go:356] "Generic (PLEG): container finished" podID="8e81c096-2704-4d9e-80b8-9b1f05e7151f" containerID="9b3267105635039f6fce14887cdffa568d8ed5d82c9b4b35612a9f9fc490dbdb" exitCode=60 Oct 30 02:30:09.122233 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.122205 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" event={"ID":"8e81c096-2704-4d9e-80b8-9b1f05e7151f","Type":"ContainerDied","Data":"9b3267105635039f6fce14887cdffa568d8ed5d82c9b4b35612a9f9fc490dbdb"} Oct 30 02:30:09.122945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.122922 5998 scope.go:122] "RemoveContainer" containerID="9b3267105635039f6fce14887cdffa568d8ed5d82c9b4b35612a9f9fc490dbdb" Oct 30 02:30:09.123455 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:09.123426 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z_odh-ai-gateway-infra(8e81c096-2704-4d9e-80b8-9b1f05e7151f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" podUID="8e81c096-2704-4d9e-80b8-9b1f05e7151f" Oct 30 02:30:09.132029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.130500 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" podStartSLOduration=9.130487762 podStartE2EDuration="9.130487762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:09.125036639 +0000 UTC m=+3638.703827720" watchObservedRunningTime="2026-10-30 02:30:09.130487762 +0000 UTC m=+3638.709278843" Oct 30 02:30:09.132432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.132403 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" event={"ID":"3f69dd9b-8eda-4368-b693-33c5b9721710","Type":"ContainerStarted","Data":"600b823011685fdd9fe8107a0fae196342b242f75ce6eee965ef31125f978a36"} Oct 30 02:30:09.135686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.135653 5998 generic.go:356] "Generic (PLEG): container finished" podID="860ba768-1529-403e-ba81-13395d6cee83" containerID="1e8cc9dcedae3a0189b91de81ab12c7c76a38fa54f0f541997685c79fda78c03" exitCode=60 Oct 30 02:30:09.135969 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.135900 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" event={"ID":"860ba768-1529-403e-ba81-13395d6cee83","Type":"ContainerDied","Data":"1e8cc9dcedae3a0189b91de81ab12c7c76a38fa54f0f541997685c79fda78c03"} Oct 30 02:30:09.136192 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.136175 5998 scope.go:122] "RemoveContainer" containerID="1e8cc9dcedae3a0189b91de81ab12c7c76a38fa54f0f541997685c79fda78c03" Oct 30 02:30:09.136590 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:09.136569 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g_odh-ai-gateway-infra(860ba768-1529-403e-ba81-13395d6cee83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" podUID="860ba768-1529-403e-ba81-13395d6cee83" Oct 30 02:30:09.141174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.141148 5998 generic.go:356] "Generic (PLEG): container finished" podID="7ff76844-137e-44ec-b146-4d9c84997ada" containerID="c2870e68e7edfdb62185f69598aea4df8767568a247e024b00e20d7dfc7c02ed" exitCode=60 Oct 30 02:30:09.141333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.141310 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" event={"ID":"7ff76844-137e-44ec-b146-4d9c84997ada","Type":"ContainerDied","Data":"c2870e68e7edfdb62185f69598aea4df8767568a247e024b00e20d7dfc7c02ed"} Oct 30 02:30:09.141563 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.141545 5998 scope.go:122] "RemoveContainer" containerID="c2870e68e7edfdb62185f69598aea4df8767568a247e024b00e20d7dfc7c02ed" Oct 30 02:30:09.141964 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:09.141935 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp_odh-ai-gateway-infra(7ff76844-137e-44ec-b146-4d9c84997ada)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" podUID="7ff76844-137e-44ec-b146-4d9c84997ada" Oct 30 02:30:09.144244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.144225 5998 generic.go:356] "Generic (PLEG): container finished" podID="1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee" containerID="a9307cc5ccab074ce26367e32ecd77e28e9306b249bb36ed62f49cdb022a85db" exitCode=60 Oct 30 02:30:09.144464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.144309 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" event={"ID":"1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee","Type":"ContainerDied","Data":"a9307cc5ccab074ce26367e32ecd77e28e9306b249bb36ed62f49cdb022a85db"} Oct 30 02:30:09.144464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.144386 5998 scope.go:122] "RemoveContainer" containerID="431bfb2cacf1d8f5eca603577dd4019e6a593c3e4cc2fbb1f61bbdd5aceccb41" Oct 30 02:30:09.144657 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.144637 5998 scope.go:122] "RemoveContainer" containerID="a9307cc5ccab074ce26367e32ecd77e28e9306b249bb36ed62f49cdb022a85db" Oct 30 02:30:09.145050 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:09.145029 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw_odh-ai-gateway-infra(1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" podUID="1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee" Oct 30 02:30:09.161282 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.161259 5998 scope.go:122] "RemoveContainer" containerID="37be78c89daf9617b3abde34f17cc4fb07f593213bfbde659bd70b302b993cf1" Oct 30 02:30:09.178577 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.178552 5998 scope.go:122] "RemoveContainer" containerID="085be5d1f60316d1b76554f1dd2bb13ee5c6d0c644c8e2f2997e129f4f54213a" Oct 30 02:30:09.188253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.188219 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:09.195323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.195296 5998 scope.go:122] "RemoveContainer" containerID="7dff198935494a5c60e7d09ae675624a72d96442b5576e39cf49744a6a724b23" Oct 30 02:30:09.325463 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.324123 5998 scope.go:122] "RemoveContainer" containerID="7ddbdd413c7ac1dcf9ccafb8874e40a89701e3c518eff89900fec74caf6b3455" Oct 30 02:30:09.364499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.364475 5998 scope.go:122] "RemoveContainer" containerID="4284dcc22fd4e42eebddee7d5ceb78fa68fa40b5c0a458646f32bc27ef538eee" Oct 30 02:30:09.387328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.387300 5998 scope.go:122] "RemoveContainer" containerID="b961458c52fddbdde3196ec9997ec8324461dab8a29006000adaf76fddf80ee2" Oct 30 02:30:09.414873 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:09.409610 5998 scope.go:122] "RemoveContainer" containerID="8b52af96be2391667b175cec58d51d6207ed530c0928bc72e6db21c3505181a1" Oct 30 02:30:10.160705 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.160673 5998 scope.go:122] "RemoveContainer" containerID="bfd51dfd12fc0d920e83de5d303f6a3ee47c919afd18e5c105fb8e82a1d9fd9d" Oct 30 02:30:10.161192 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.161129 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt_odh-ai-gateway-infra(a105635a-7115-4c02-aa79-60516391579b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" podUID="a105635a-7115-4c02-aa79-60516391579b" Oct 30 02:30:10.162741 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.162714 5998 scope.go:122] "RemoveContainer" containerID="9b3267105635039f6fce14887cdffa568d8ed5d82c9b4b35612a9f9fc490dbdb" Oct 30 02:30:10.163149 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.163121 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z_odh-ai-gateway-infra(8e81c096-2704-4d9e-80b8-9b1f05e7151f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" podUID="8e81c096-2704-4d9e-80b8-9b1f05e7151f" Oct 30 02:30:10.164502 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.164477 5998 generic.go:356] "Generic (PLEG): container finished" podID="3f69dd9b-8eda-4368-b693-33c5b9721710" containerID="600b823011685fdd9fe8107a0fae196342b242f75ce6eee965ef31125f978a36" exitCode=60 Oct 30 02:30:10.164611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.164562 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" event={"ID":"3f69dd9b-8eda-4368-b693-33c5b9721710","Type":"ContainerDied","Data":"600b823011685fdd9fe8107a0fae196342b242f75ce6eee965ef31125f978a36"} Oct 30 02:30:10.164669 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.164620 5998 scope.go:122] "RemoveContainer" containerID="6fc4f0974222bd2edb4a418038e65eb9d9173a437a01710f4b91ef2c2d89333d" Oct 30 02:30:10.164890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.164863 5998 scope.go:122] "RemoveContainer" containerID="600b823011685fdd9fe8107a0fae196342b242f75ce6eee965ef31125f978a36" Oct 30 02:30:10.165340 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.165316 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh_odh-ai-gateway-infra(3f69dd9b-8eda-4368-b693-33c5b9721710)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" podUID="3f69dd9b-8eda-4368-b693-33c5b9721710" Oct 30 02:30:10.167694 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.167380 5998 scope.go:122] "RemoveContainer" containerID="1e8cc9dcedae3a0189b91de81ab12c7c76a38fa54f0f541997685c79fda78c03" Oct 30 02:30:10.167794 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.167742 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g_odh-ai-gateway-infra(860ba768-1529-403e-ba81-13395d6cee83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" podUID="860ba768-1529-403e-ba81-13395d6cee83" Oct 30 02:30:10.169218 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.169195 5998 scope.go:122] "RemoveContainer" containerID="c2870e68e7edfdb62185f69598aea4df8767568a247e024b00e20d7dfc7c02ed" Oct 30 02:30:10.169564 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.169538 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp_odh-ai-gateway-infra(7ff76844-137e-44ec-b146-4d9c84997ada)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" podUID="7ff76844-137e-44ec-b146-4d9c84997ada" Oct 30 02:30:10.171163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.171141 5998 scope.go:122] "RemoveContainer" containerID="a9307cc5ccab074ce26367e32ecd77e28e9306b249bb36ed62f49cdb022a85db" Oct 30 02:30:10.171497 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.171469 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw_odh-ai-gateway-infra(1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" podUID="1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee" Oct 30 02:30:10.173306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.173276 5998 generic.go:356] "Generic (PLEG): container finished" podID="c9b0ea41-0e61-456d-bbce-6ee7145552ea" containerID="e3cd7f2ab448a9fb591dcb3c307bc71ea1ee8373a79b61aa5a92080b6704e74e" exitCode=60 Oct 30 02:30:10.173453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.173428 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" event={"ID":"c9b0ea41-0e61-456d-bbce-6ee7145552ea","Type":"ContainerDied","Data":"e3cd7f2ab448a9fb591dcb3c307bc71ea1ee8373a79b61aa5a92080b6704e74e"} Oct 30 02:30:10.173643 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.173627 5998 scope.go:122] "RemoveContainer" containerID="e3cd7f2ab448a9fb591dcb3c307bc71ea1ee8373a79b61aa5a92080b6704e74e" Oct 30 02:30:10.173924 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.173901 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp_odh-ai-gateway-infra(c9b0ea41-0e61-456d-bbce-6ee7145552ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" podUID="c9b0ea41-0e61-456d-bbce-6ee7145552ea" Oct 30 02:30:10.176587 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.176563 5998 generic.go:356] "Generic (PLEG): container finished" podID="046ebf9b-3d7e-4c67-bfcf-8545980b59ff" containerID="06ba2ce69cb9eb1821fef5fdb771d2512a2d3740b31562f876c764ccc8cec9bc" exitCode=60 Oct 30 02:30:10.176695 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.176630 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" event={"ID":"046ebf9b-3d7e-4c67-bfcf-8545980b59ff","Type":"ContainerDied","Data":"06ba2ce69cb9eb1821fef5fdb771d2512a2d3740b31562f876c764ccc8cec9bc"} Oct 30 02:30:10.176987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.176954 5998 scope.go:122] "RemoveContainer" containerID="06ba2ce69cb9eb1821fef5fdb771d2512a2d3740b31562f876c764ccc8cec9bc" Oct 30 02:30:10.177343 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.177308 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn_odh-ai-gateway-infra(046ebf9b-3d7e-4c67-bfcf-8545980b59ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" podUID="046ebf9b-3d7e-4c67-bfcf-8545980b59ff" Oct 30 02:30:10.179748 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.179718 5998 scope.go:122] "RemoveContainer" containerID="e8c11b0e22deafb51a20cb90ebd0266e9fe0759616e63a29e7ec5eb0176f6462" Oct 30 02:30:10.180045 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.180022 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf_odh-ai-gateway-infra(be309ab3-73d7-42c0-92a8-0fc686504795)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" podUID="be309ab3-73d7-42c0-92a8-0fc686504795" Oct 30 02:30:10.181658 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.181636 5998 scope.go:122] "RemoveContainer" containerID="ef3e023f57ad7986d43f1756955746298bd01f8d4ec96b43589aa582cdb699b1" Oct 30 02:30:10.181980 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.181957 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd_odh-ai-gateway-infra(a1f42aec-2169-46f1-a837-039b920ac828)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" podUID="a1f42aec-2169-46f1-a837-039b920ac828" Oct 30 02:30:10.183802 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.183779 5998 generic.go:356] "Generic (PLEG): container finished" podID="5bce6d68-142e-46ef-b408-89e44ca0fc10" containerID="edb54ca7e39dadc76acc72f712f3eff40a4e350b7ab8a7cde2b6cfa9b6634ee2" exitCode=60 Oct 30 02:30:10.183910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.183859 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" event={"ID":"5bce6d68-142e-46ef-b408-89e44ca0fc10","Type":"ContainerDied","Data":"edb54ca7e39dadc76acc72f712f3eff40a4e350b7ab8a7cde2b6cfa9b6634ee2"} Oct 30 02:30:10.184139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.184124 5998 scope.go:122] "RemoveContainer" containerID="edb54ca7e39dadc76acc72f712f3eff40a4e350b7ab8a7cde2b6cfa9b6634ee2" Oct 30 02:30:10.184365 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.184348 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k_odh-ai-gateway-infra(5bce6d68-142e-46ef-b408-89e44ca0fc10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" podUID="5bce6d68-142e-46ef-b408-89e44ca0fc10" Oct 30 02:30:10.189766 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.189744 5998 generic.go:356] "Generic (PLEG): container finished" podID="96bc88e1-7b35-4b54-b12f-69227ae41a88" containerID="fb366deaa1e6af17dc5d8d44b5a2453a45bd6924de38dc3a5519c08c7e33f769" exitCode=60 Oct 30 02:30:10.189866 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.189815 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" event={"ID":"96bc88e1-7b35-4b54-b12f-69227ae41a88","Type":"ContainerDied","Data":"fb366deaa1e6af17dc5d8d44b5a2453a45bd6924de38dc3a5519c08c7e33f769"} Oct 30 02:30:10.190134 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.190081 5998 scope.go:122] "RemoveContainer" containerID="fb366deaa1e6af17dc5d8d44b5a2453a45bd6924de38dc3a5519c08c7e33f769" Oct 30 02:30:10.190476 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.190455 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj_odh-ai-gateway-infra(96bc88e1-7b35-4b54-b12f-69227ae41a88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" podUID="96bc88e1-7b35-4b54-b12f-69227ae41a88" Oct 30 02:30:10.191759 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.191732 5998 generic.go:356] "Generic (PLEG): container finished" podID="2f3ead60-88ca-42af-9e25-7822589058e4" containerID="beeb1625a64a57cc79961b5b7fa1df2ceae6b5c926f85462418aa49c50a11b81" exitCode=60 Oct 30 02:30:10.191836 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.191774 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" event={"ID":"2f3ead60-88ca-42af-9e25-7822589058e4","Type":"ContainerDied","Data":"beeb1625a64a57cc79961b5b7fa1df2ceae6b5c926f85462418aa49c50a11b81"} Oct 30 02:30:10.192168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.192148 5998 scope.go:122] "RemoveContainer" containerID="beeb1625a64a57cc79961b5b7fa1df2ceae6b5c926f85462418aa49c50a11b81" Oct 30 02:30:10.192526 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.192500 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs_odh-ai-gateway-infra(2f3ead60-88ca-42af-9e25-7822589058e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" podUID="2f3ead60-88ca-42af-9e25-7822589058e4" Oct 30 02:30:10.193799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.193779 5998 scope.go:122] "RemoveContainer" containerID="8ef843f3d22e1c1029643acdb1e32dfea707800156c7a4cd6d1b284c6c5b5637" Oct 30 02:30:10.194130 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.194071 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn_odh-ai-gateway-infra(3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" podUID="3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb" Oct 30 02:30:10.195599 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.195577 5998 generic.go:356] "Generic (PLEG): container finished" podID="0771e950-74b9-45cb-a7be-ccb30c8c8c65" containerID="79d0c281cd74ee8e1752f6542df043055bcfeb5166ad788019792279e227aca4" exitCode=60 Oct 30 02:30:10.195714 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.195650 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" event={"ID":"0771e950-74b9-45cb-a7be-ccb30c8c8c65","Type":"ContainerDied","Data":"79d0c281cd74ee8e1752f6542df043055bcfeb5166ad788019792279e227aca4"} Oct 30 02:30:10.195878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.195864 5998 scope.go:122] "RemoveContainer" containerID="79d0c281cd74ee8e1752f6542df043055bcfeb5166ad788019792279e227aca4" Oct 30 02:30:10.196167 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.196140 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf_odh-ai-gateway-infra(0771e950-74b9-45cb-a7be-ccb30c8c8c65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" podUID="0771e950-74b9-45cb-a7be-ccb30c8c8c65" Oct 30 02:30:10.197643 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.197614 5998 generic.go:356] "Generic (PLEG): container finished" podID="512d72f9-4883-4d39-9df8-9282f56da2ac" containerID="b947ecaf282ade6974160eab1ab8046a36d627d04da067735804d90d1d0613ef" exitCode=60 Oct 30 02:30:10.197744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.197694 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" event={"ID":"512d72f9-4883-4d39-9df8-9282f56da2ac","Type":"ContainerDied","Data":"b947ecaf282ade6974160eab1ab8046a36d627d04da067735804d90d1d0613ef"} Oct 30 02:30:10.197917 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.197902 5998 scope.go:122] "RemoveContainer" containerID="b947ecaf282ade6974160eab1ab8046a36d627d04da067735804d90d1d0613ef" Oct 30 02:30:10.198238 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.198212 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr_odh-ai-gateway-infra(512d72f9-4883-4d39-9df8-9282f56da2ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" podUID="512d72f9-4883-4d39-9df8-9282f56da2ac" Oct 30 02:30:10.200066 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.200047 5998 generic.go:356] "Generic (PLEG): container finished" podID="9e4909d0-f3ae-423a-a478-4ee5ce9734a1" containerID="bfe24a95fde280b39bfc982bd812c048dbe52ef1d7489db9b9f45ea694d55eb1" exitCode=60 Oct 30 02:30:10.200179 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.200079 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" event={"ID":"9e4909d0-f3ae-423a-a478-4ee5ce9734a1","Type":"ContainerDied","Data":"bfe24a95fde280b39bfc982bd812c048dbe52ef1d7489db9b9f45ea694d55eb1"} Oct 30 02:30:10.200491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.200468 5998 scope.go:122] "RemoveContainer" containerID="bfe24a95fde280b39bfc982bd812c048dbe52ef1d7489db9b9f45ea694d55eb1" Oct 30 02:30:10.200910 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.200867 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc_odh-ai-gateway-infra(9e4909d0-f3ae-423a-a478-4ee5ce9734a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" podUID="9e4909d0-f3ae-423a-a478-4ee5ce9734a1" Oct 30 02:30:10.204242 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.204219 5998 scope.go:122] "RemoveContainer" containerID="1e12b2bcf54976d519b107ab2a34702c508caeb6f5ebc61b072d92a53e171474" Oct 30 02:30:10.204624 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.204590 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6_odh-ai-gateway-infra(e5882ea9-fe2e-4587-abb7-bb14939f53f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" podUID="e5882ea9-fe2e-4587-abb7-bb14939f53f6" Oct 30 02:30:10.206861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.206835 5998 generic.go:356] "Generic (PLEG): container finished" podID="daaa2b88-a49a-4a88-8325-33307e969074" containerID="4808839e8ecf47d51616536ac0709d0b7dff0daac511e5a59c6ce6f9dbe81f12" exitCode=60 Oct 30 02:30:10.206990 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.206973 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" event={"ID":"daaa2b88-a49a-4a88-8325-33307e969074","Type":"ContainerDied","Data":"4808839e8ecf47d51616536ac0709d0b7dff0daac511e5a59c6ce6f9dbe81f12"} Oct 30 02:30:10.207475 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.207456 5998 scope.go:122] "RemoveContainer" containerID="4808839e8ecf47d51616536ac0709d0b7dff0daac511e5a59c6ce6f9dbe81f12" Oct 30 02:30:10.207775 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.207745 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2_odh-ai-gateway-infra(daaa2b88-a49a-4a88-8325-33307e969074)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" podUID="daaa2b88-a49a-4a88-8325-33307e969074" Oct 30 02:30:10.210220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.210193 5998 generic.go:356] "Generic (PLEG): container finished" podID="115cd7ff-80aa-4cd9-a247-9154fe7d117f" containerID="55cba0eb40314867cf6b2d66f160469b841bb0c32284f6987eafd0778727b966" exitCode=60 Oct 30 02:30:10.210316 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.210239 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" event={"ID":"115cd7ff-80aa-4cd9-a247-9154fe7d117f","Type":"ContainerDied","Data":"55cba0eb40314867cf6b2d66f160469b841bb0c32284f6987eafd0778727b966"} Oct 30 02:30:10.210629 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.210606 5998 scope.go:122] "RemoveContainer" containerID="55cba0eb40314867cf6b2d66f160469b841bb0c32284f6987eafd0778727b966" Oct 30 02:30:10.210896 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:10.210873 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff_odh-ai-gateway-infra(115cd7ff-80aa-4cd9-a247-9154fe7d117f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" podUID="115cd7ff-80aa-4cd9-a247-9154fe7d117f" Oct 30 02:30:10.321479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.321450 5998 scope.go:122] "RemoveContainer" containerID="dd1f75e17ef793719a2a64626fbd646480e9a7f8f29bf8b13a785bb78b9fbdc6" Oct 30 02:30:10.339946 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.339921 5998 scope.go:122] "RemoveContainer" containerID="86743c3c7f633eaaa1a175d40d1c3cdaa4dc80bf86dc193a59fe5fd8ca94f969" Oct 30 02:30:10.360446 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.359103 5998 scope.go:122] "RemoveContainer" containerID="4899e73bb09c44c832db82e0c44fcd76aa34557cb93cc9f03819fc52e6a483e2" Oct 30 02:30:10.384708 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.384679 5998 scope.go:122] "RemoveContainer" containerID="c8ad237c55c76c7ef258fc7442b4cf713f2d11f4b65cabb9b27679a9d8164a4e" Oct 30 02:30:10.403325 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.403299 5998 scope.go:122] "RemoveContainer" containerID="73ec7df62572677493e234e0dc20d5a86b06f7ec9019e85d14a51f436d0f2425" Oct 30 02:30:10.421454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.421424 5998 scope.go:122] "RemoveContainer" containerID="da39fef5cb879a319437b3c2f9cede8e39e485c9f0ede3d651eba984fc314a5a" Oct 30 02:30:10.437872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.437843 5998 scope.go:122] "RemoveContainer" containerID="4d235597540bf96806bdc9dd108132d5f98a035aba4ffabadbd717d9d1119a71" Oct 30 02:30:10.458886 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.458856 5998 scope.go:122] "RemoveContainer" containerID="0a89aced9636173d35bda68cce1b6bb74220865730c332ddc9db79422bc26189" Oct 30 02:30:10.476194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.476170 5998 scope.go:122] "RemoveContainer" containerID="22c3a3fbb220d8eca90cab8b3bdff426bfec46a1e2292cebb40368ffff31aa83" Oct 30 02:30:10.492709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.492686 5998 scope.go:122] "RemoveContainer" containerID="c93f61213e974218a97064a9061f11f0869a78e8227079a6743ea373859cd085" Oct 30 02:30:10.779965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:10.779914 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:11.226657 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:11.226626 5998 scope.go:122] "RemoveContainer" containerID="55cba0eb40314867cf6b2d66f160469b841bb0c32284f6987eafd0778727b966" Oct 30 02:30:11.227225 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:11.227172 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff_odh-ai-gateway-infra(115cd7ff-80aa-4cd9-a247-9154fe7d117f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" podUID="115cd7ff-80aa-4cd9-a247-9154fe7d117f" Oct 30 02:30:11.228892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:11.228864 5998 scope.go:122] "RemoveContainer" containerID="600b823011685fdd9fe8107a0fae196342b242f75ce6eee965ef31125f978a36" Oct 30 02:30:11.229424 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:11.229263 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh_odh-ai-gateway-infra(3f69dd9b-8eda-4368-b693-33c5b9721710)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" podUID="3f69dd9b-8eda-4368-b693-33c5b9721710" Oct 30 02:30:11.231344 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:11.231323 5998 scope.go:122] "RemoveContainer" containerID="e3cd7f2ab448a9fb591dcb3c307bc71ea1ee8373a79b61aa5a92080b6704e74e" Oct 30 02:30:11.231860 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:11.231816 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp_odh-ai-gateway-infra(c9b0ea41-0e61-456d-bbce-6ee7145552ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" podUID="c9b0ea41-0e61-456d-bbce-6ee7145552ea" Oct 30 02:30:11.233482 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:11.233442 5998 scope.go:122] "RemoveContainer" containerID="06ba2ce69cb9eb1821fef5fdb771d2512a2d3740b31562f876c764ccc8cec9bc" Oct 30 02:30:11.233808 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:11.233785 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn_odh-ai-gateway-infra(046ebf9b-3d7e-4c67-bfcf-8545980b59ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" podUID="046ebf9b-3d7e-4c67-bfcf-8545980b59ff" Oct 30 02:30:11.236316 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:11.236290 5998 scope.go:122] "RemoveContainer" containerID="edb54ca7e39dadc76acc72f712f3eff40a4e350b7ab8a7cde2b6cfa9b6634ee2" Oct 30 02:30:11.236631 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:11.236608 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k_odh-ai-gateway-infra(5bce6d68-142e-46ef-b408-89e44ca0fc10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" podUID="5bce6d68-142e-46ef-b408-89e44ca0fc10" Oct 30 02:30:11.238923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:11.238900 5998 scope.go:122] "RemoveContainer" containerID="fb366deaa1e6af17dc5d8d44b5a2453a45bd6924de38dc3a5519c08c7e33f769" Oct 30 02:30:11.239289 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:11.239264 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj_odh-ai-gateway-infra(96bc88e1-7b35-4b54-b12f-69227ae41a88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" podUID="96bc88e1-7b35-4b54-b12f-69227ae41a88" Oct 30 02:30:11.243928 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:11.243905 5998 scope.go:122] "RemoveContainer" containerID="beeb1625a64a57cc79961b5b7fa1df2ceae6b5c926f85462418aa49c50a11b81" Oct 30 02:30:11.244288 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:11.244260 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs_odh-ai-gateway-infra(2f3ead60-88ca-42af-9e25-7822589058e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" podUID="2f3ead60-88ca-42af-9e25-7822589058e4" Oct 30 02:30:11.247268 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:11.247245 5998 scope.go:122] "RemoveContainer" containerID="79d0c281cd74ee8e1752f6542df043055bcfeb5166ad788019792279e227aca4" Oct 30 02:30:11.247523 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:11.247502 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf_odh-ai-gateway-infra(0771e950-74b9-45cb-a7be-ccb30c8c8c65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" podUID="0771e950-74b9-45cb-a7be-ccb30c8c8c65" Oct 30 02:30:11.249414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:11.249333 5998 scope.go:122] "RemoveContainer" containerID="b947ecaf282ade6974160eab1ab8046a36d627d04da067735804d90d1d0613ef" Oct 30 02:30:11.249653 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:11.249627 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr_odh-ai-gateway-infra(512d72f9-4883-4d39-9df8-9282f56da2ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" podUID="512d72f9-4883-4d39-9df8-9282f56da2ac" Oct 30 02:30:11.251223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:11.251203 5998 scope.go:122] "RemoveContainer" containerID="bfe24a95fde280b39bfc982bd812c048dbe52ef1d7489db9b9f45ea694d55eb1" Oct 30 02:30:11.251511 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:11.251483 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc_odh-ai-gateway-infra(9e4909d0-f3ae-423a-a478-4ee5ce9734a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" podUID="9e4909d0-f3ae-423a-a478-4ee5ce9734a1" Oct 30 02:30:11.253053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:11.253036 5998 scope.go:122] "RemoveContainer" containerID="4808839e8ecf47d51616536ac0709d0b7dff0daac511e5a59c6ce6f9dbe81f12" Oct 30 02:30:11.253287 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:11.253269 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2_odh-ai-gateway-infra(daaa2b88-a49a-4a88-8325-33307e969074)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" podUID="daaa2b88-a49a-4a88-8325-33307e969074" Oct 30 02:30:12.392839 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:12.392795 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:12.652667 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:12.652579 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:13.177295 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:13.177252 5998 patch_prober.go:29] interesting pod/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:30:13.177508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:13.177329 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" podUID="cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:30:14.132111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:14.132047 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:14.284247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:14.284214 5998 generic.go:356] "Generic (PLEG): container finished" podID="cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" containerID="c1eaa521288148421bc1583339bf04077523312cebfc7375f7db69d923bdcfa5" exitCode=0 Oct 30 02:30:14.284433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:14.284289 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" event={"ID":"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c","Type":"ContainerDied","Data":"c1eaa521288148421bc1583339bf04077523312cebfc7375f7db69d923bdcfa5"} Oct 30 02:30:14.284853 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:14.284820 5998 scope.go:122] "RemoveContainer" containerID="c1eaa521288148421bc1583339bf04077523312cebfc7375f7db69d923bdcfa5" Oct 30 02:30:14.287387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:14.287351 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:30:14.287485 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:14.287428 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:30:14.287485 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:14.287475 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:30:15.297989 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:15.297945 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" event={"ID":"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c","Type":"ContainerStarted","Data":"953e0b47c15e8a44542b8d2abafc674f8aa8204d722fce31498c13b239e8f3ca"} Oct 30 02:30:15.583921 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:15.583819 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:15.998878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:15.998843 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:30:15.999167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:15.999136 5998 patch_prober.go:29] interesting pod/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Oct 30 02:30:15.999285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:15.999207 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" podUID="cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Oct 30 02:30:16.999325 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:16.999285 5998 patch_prober.go:29] interesting pod/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Oct 30 02:30:16.999831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:16.999387 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" podUID="cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Oct 30 02:30:17.039984 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:17.039940 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj"] Oct 30 02:30:17.049420 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:17.049380 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11f0a78f-wvzjj"] Oct 30 02:30:17.293601 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:17.293476 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:17.998788 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:17.998753 5998 patch_prober.go:29] interesting pod/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Oct 30 02:30:17.999083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:17.999049 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" podUID="cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Oct 30 02:30:18.046261 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:18.046195 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452"] Oct 30 02:30:18.052166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:18.052126 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f255f8f-tv452"] Oct 30 02:30:18.536702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:18.536657 5998 scope.go:122] "RemoveContainer" containerID="c2870e68e7edfdb62185f69598aea4df8767568a247e024b00e20d7dfc7c02ed" Oct 30 02:30:18.560098 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:18.559410 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed" path="/var/lib/kubelet/pods/d3ae4d4a-eca4-4ddb-a316-9d9ab9598bed/volumes" Oct 30 02:30:18.561127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:18.560300 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb9e882-4da2-4472-95ae-84ac0d5d419e" path="/var/lib/kubelet/pods/fbb9e882-4da2-4472-95ae-84ac0d5d419e/volumes" Oct 30 02:30:19.006732 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:19.006691 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:19.009543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:19.009515 5998 patch_prober.go:29] interesting pod/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Oct 30 02:30:19.009780 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:19.009751 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" podUID="cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Oct 30 02:30:19.444113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:19.443878 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:19.473987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:19.473956 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" event={"ID":"7ff76844-137e-44ec-b146-4d9c84997ada","Type":"ContainerStarted","Data":"86e19a0609c901c6728c0edb6c5b01b29a128bb1933e6313f086b43ac7acf357"} Oct 30 02:30:19.545167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:19.544935 5998 scope.go:122] "RemoveContainer" containerID="1555edc8ec84f776cd24750e285b0d3a85649a39e915c06462a76670d55d26e0" Oct 30 02:30:19.545340 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:19.545228 5998 scope.go:122] "RemoveContainer" containerID="e8c11b0e22deafb51a20cb90ebd0266e9fe0759616e63a29e7ec5eb0176f6462" Oct 30 02:30:19.570125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:19.546054 5998 scope.go:122] "RemoveContainer" containerID="1e8cc9dcedae3a0189b91de81ab12c7c76a38fa54f0f541997685c79fda78c03" Oct 30 02:30:19.570125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:19.546918 5998 scope.go:122] "RemoveContainer" containerID="a9307cc5ccab074ce26367e32ecd77e28e9306b249bb36ed62f49cdb022a85db" Oct 30 02:30:19.576110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:19.571857 5998 scope.go:122] "RemoveContainer" containerID="9b3267105635039f6fce14887cdffa568d8ed5d82c9b4b35612a9f9fc490dbdb" Oct 30 02:30:19.576110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:19.573782 5998 scope.go:122] "RemoveContainer" containerID="ef3e023f57ad7986d43f1756955746298bd01f8d4ec96b43589aa582cdb699b1" Oct 30 02:30:19.591291 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:19.591261 5998 scope.go:122] "RemoveContainer" containerID="bfd51dfd12fc0d920e83de5d303f6a3ee47c919afd18e5c105fb8e82a1d9fd9d" Oct 30 02:30:19.939114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:19.934803 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:19.999630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:19.998901 5998 patch_prober.go:29] interesting pod/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Oct 30 02:30:19.999630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:19.998949 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" podUID="cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Oct 30 02:30:20.423218 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:20.415775 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:20.770136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:20.752078 5998 scope.go:122] "RemoveContainer" containerID="06ba2ce69cb9eb1821fef5fdb771d2512a2d3740b31562f876c764ccc8cec9bc" Oct 30 02:30:20.826114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:20.785415 5998 scope.go:122] "RemoveContainer" containerID="600b823011685fdd9fe8107a0fae196342b242f75ce6eee965ef31125f978a36" Oct 30 02:30:21.104963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.104929 5998 patch_prober.go:29] interesting pod/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Oct 30 02:30:21.105313 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.105278 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" podUID="cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Oct 30 02:30:21.177879 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.177723 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:21.360135 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.350595 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx"] Oct 30 02:30:21.467192 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.467135 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 02:30:21.468138 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.468070 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:21.473567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.473061 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:30:21.544696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.543453 5998 generic.go:356] "Generic (PLEG): container finished" podID="7ff76844-137e-44ec-b146-4d9c84997ada" containerID="86e19a0609c901c6728c0edb6c5b01b29a128bb1933e6313f086b43ac7acf357" exitCode=60 Oct 30 02:30:21.544696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.543536 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx"] Oct 30 02:30:21.544696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.543579 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp"] Oct 30 02:30:21.544696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.543606 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" event={"ID":"7ff76844-137e-44ec-b146-4d9c84997ada","Type":"ContainerDied","Data":"86e19a0609c901c6728c0edb6c5b01b29a128bb1933e6313f086b43ac7acf357"} Oct 30 02:30:21.544696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.543638 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5"] Oct 30 02:30:21.544696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.543705 5998 scope.go:122] "RemoveContainer" containerID="c2870e68e7edfdb62185f69598aea4df8767568a247e024b00e20d7dfc7c02ed" Oct 30 02:30:21.545788 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.545316 5998 scope.go:122] "RemoveContainer" containerID="b947ecaf282ade6974160eab1ab8046a36d627d04da067735804d90d1d0613ef" Oct 30 02:30:21.547113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.546236 5998 scope.go:122] "RemoveContainer" containerID="26c7ffac65a2098d82f0c88a36285c9676a5074e2def1c037e1b04c0bba86ef0" Oct 30 02:30:21.547113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.546824 5998 scope.go:122] "RemoveContainer" containerID="8ef843f3d22e1c1029643acdb1e32dfea707800156c7a4cd6d1b284c6c5b5637" Oct 30 02:30:21.549821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.548687 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.549821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.549674 5998 scope.go:122] "RemoveContainer" containerID="1e12b2bcf54976d519b107ab2a34702c508caeb6f5ebc61b072d92a53e171474" Oct 30 02:30:21.573934 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.571483 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-07fb25-openshift-default-dockercfg-8p8fw\"" Oct 30 02:30:21.718305 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.718270 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5"] Oct 30 02:30:21.718464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.718446 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.736434 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.736406 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3d5da6-openshift-default-dockercfg-t75nk\"" Oct 30 02:30:21.800534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.796687 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-podinfo\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.800534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.796777 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istiod-ca-cert\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.800534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.796829 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-token\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.800534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.796920 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-workload-certs\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.800534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.796973 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-credential-socket\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.800534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.796999 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vw87\" (UniqueName: \"kubernetes.io/projected/1d926a5c-800a-4b12-b134-be5c4ff7a619-kube-api-access-8vw87\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.800534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.797057 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2lxx\" (UniqueName: \"kubernetes.io/projected/1dac6d55-be03-4ce9-b8a7-a6397747e73d-kube-api-access-w2lxx\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.800534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.797116 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d926a5c-800a-4b12-b134-be5c4ff7a619-istiod-ca-cert\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.800534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.797165 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-workload-socket\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.800534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.797213 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-podinfo\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.800534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.797251 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-credential-socket\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.800534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.797330 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-workload-socket\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.800534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.797366 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-token\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.800534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.797410 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-envoy\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.800534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.797436 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-data\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.800534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.797459 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-data\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.800534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.797501 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-envoy\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.800534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.797599 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-workload-certs\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.899769 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.899727 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-workload-certs\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.899961 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.899796 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-podinfo\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.899961 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.899868 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istiod-ca-cert\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.899961 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.899920 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-token\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.900220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.900019 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-workload-certs\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.900220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.900075 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-credential-socket\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.900220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.900159 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vw87\" (UniqueName: \"kubernetes.io/projected/1d926a5c-800a-4b12-b134-be5c4ff7a619-kube-api-access-8vw87\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.900390 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.900228 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2lxx\" (UniqueName: \"kubernetes.io/projected/1dac6d55-be03-4ce9-b8a7-a6397747e73d-kube-api-access-w2lxx\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.900390 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.900284 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d926a5c-800a-4b12-b134-be5c4ff7a619-istiod-ca-cert\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.900390 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.900344 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-workload-socket\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.900549 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.900399 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-podinfo\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.900549 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.900442 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-credential-socket\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.900549 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.900542 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-workload-socket\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.900713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.900584 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-token\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.900713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.900638 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-envoy\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.900713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.900670 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-data\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.900713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.900695 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-data\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.900911 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.900746 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-envoy\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.905492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.904552 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-workload-certs\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.905492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.905115 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-workload-socket\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.905492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.905239 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d926a5c-800a-4b12-b134-be5c4ff7a619-istiod-ca-cert\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.907016 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.906231 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-credential-socket\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.907016 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.906514 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-workload-socket\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.907016 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.906756 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-data\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.913745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.911378 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-workload-certs\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.913745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.911997 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istiod-ca-cert\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.913745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.912709 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-credential-socket\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.913745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.913170 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-data\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.923876 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.923813 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-podinfo\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:21.928982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.928953 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-envoy\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:21.931716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:21.931686 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-podinfo\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:22.012812 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.007837 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2lxx\" (UniqueName: \"kubernetes.io/projected/1dac6d55-be03-4ce9-b8a7-a6397747e73d-kube-api-access-w2lxx\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:22.012812 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.008080 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-envoy\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:22.012812 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.008455 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vw87\" (UniqueName: \"kubernetes.io/projected/1d926a5c-800a-4b12-b134-be5c4ff7a619-kube-api-access-8vw87\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:22.012812 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.008485 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-token\") pod \"e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:22.012812 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.012203 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-token\") pod \"e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:22.058545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.058463 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" Oct 30 02:30:22.119189 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.115694 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:30:22.124507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.120504 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:30:22.128779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.126930 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:30:22.220925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.220874 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ff76844-137e-44ec-b146-4d9c84997ada-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff76844-137e-44ec-b146-4d9c84997ada-maas-api-service-ca\") pod \"7ff76844-137e-44ec-b146-4d9c84997ada\" (UID: \"7ff76844-137e-44ec-b146-4d9c84997ada\") " Oct 30 02:30:22.221215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.221082 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff76844-137e-44ec-b146-4d9c84997ada-kube-api-access-2n8xl\" (UniqueName: \"kubernetes.io/projected/7ff76844-137e-44ec-b146-4d9c84997ada-kube-api-access-2n8xl\") pod \"7ff76844-137e-44ec-b146-4d9c84997ada\" (UID: \"7ff76844-137e-44ec-b146-4d9c84997ada\") " Oct 30 02:30:22.222855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.222824 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff76844-137e-44ec-b146-4d9c84997ada-maas-api-service-ca" pod "7ff76844-137e-44ec-b146-4d9c84997ada" (UID: "7ff76844-137e-44ec-b146-4d9c84997ada"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:22.235981 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.235947 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:22.243585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.243465 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:22.324001 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.323906 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff76844-137e-44ec-b146-4d9c84997ada-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:22.339686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.338714 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff76844-137e-44ec-b146-4d9c84997ada-kube-api-access-2n8xl" pod "7ff76844-137e-44ec-b146-4d9c84997ada" (UID: "7ff76844-137e-44ec-b146-4d9c84997ada"). InnerVolumeSpecName "kube-api-access-2n8xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:22.425650 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.425605 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n8xl\" (UniqueName: \"kubernetes.io/projected/7ff76844-137e-44ec-b146-4d9c84997ada-kube-api-access-2n8xl\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:22.545725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.543917 5998 scope.go:122] "RemoveContainer" containerID="27c4826c0628cf90c7f034eec6af68694bbe65e42077d4f46bf99fa840712636" Oct 30 02:30:22.545725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.544726 5998 scope.go:122] "RemoveContainer" containerID="edb54ca7e39dadc76acc72f712f3eff40a4e350b7ab8a7cde2b6cfa9b6634ee2" Oct 30 02:30:22.793072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.793033 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx"] Oct 30 02:30:22.794149 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:30:22.794060 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dac6d55_be03_4ce9_b8a7_a6397747e73d.slice/crio-8ca010333abe3facf8128abed95933ea85cf83b7d9fbaf8a809dd12034540c18 WatchSource:0}: Error finding container 8ca010333abe3facf8128abed95933ea85cf83b7d9fbaf8a809dd12034540c18: Status 404 returned error can't find the container with id 8ca010333abe3facf8128abed95933ea85cf83b7d9fbaf8a809dd12034540c18 Oct 30 02:30:22.799037 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.798994 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:30:22.799181 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.799101 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:30:22.799181 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.799154 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:30:22.804182 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.804150 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5"] Oct 30 02:30:22.822511 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:30:22.822478 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d926a5c_800a_4b12_b134_be5c4ff7a619.slice/crio-800a41e1bc5c1a449ad452dd347231c7487b5ea1117bf2c14431053e9be5de72 WatchSource:0}: Error finding container 800a41e1bc5c1a449ad452dd347231c7487b5ea1117bf2c14431053e9be5de72: Status 404 returned error can't find the container with id 800a41e1bc5c1a449ad452dd347231c7487b5ea1117bf2c14431053e9be5de72 Oct 30 02:30:22.848359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.848312 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:30:22.848503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.848413 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:30:22.848503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.848460 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:30:22.872533 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.872206 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:22.874401 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.873454 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" event={"ID":"046ebf9b-3d7e-4c67-bfcf-8545980b59ff","Type":"ContainerStarted","Data":"f6bc20c6b2a9cfc61514e76a54e268545b5283321c581dc3598fe5747552d2dd"} Oct 30 02:30:22.879459 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.879356 5998 generic.go:356] "Generic (PLEG): container finished" podID="be309ab3-73d7-42c0-92a8-0fc686504795" containerID="17635562922cb55f8bfa8158240dfd3128372b51d77d25d3bcf6b7c573960e14" exitCode=60 Oct 30 02:30:22.879459 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.879400 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" event={"ID":"be309ab3-73d7-42c0-92a8-0fc686504795","Type":"ContainerDied","Data":"17635562922cb55f8bfa8158240dfd3128372b51d77d25d3bcf6b7c573960e14"} Oct 30 02:30:22.879718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.879472 5998 scope.go:122] "RemoveContainer" containerID="e8c11b0e22deafb51a20cb90ebd0266e9fe0759616e63a29e7ec5eb0176f6462" Oct 30 02:30:22.879972 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.879951 5998 scope.go:122] "RemoveContainer" containerID="17635562922cb55f8bfa8158240dfd3128372b51d77d25d3bcf6b7c573960e14" Oct 30 02:30:22.880805 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:22.880435 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf_odh-ai-gateway-infra(be309ab3-73d7-42c0-92a8-0fc686504795)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" podUID="be309ab3-73d7-42c0-92a8-0fc686504795" Oct 30 02:30:22.887208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.885113 5998 generic.go:356] "Generic (PLEG): container finished" podID="a1f42aec-2169-46f1-a837-039b920ac828" containerID="a60df0e16ddcec6e2600fd4cdf65aa885b533807abbcbf37006819498af95364" exitCode=60 Oct 30 02:30:22.887208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.885273 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" event={"ID":"a1f42aec-2169-46f1-a837-039b920ac828","Type":"ContainerDied","Data":"a60df0e16ddcec6e2600fd4cdf65aa885b533807abbcbf37006819498af95364"} Oct 30 02:30:22.887208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.885767 5998 scope.go:122] "RemoveContainer" containerID="a60df0e16ddcec6e2600fd4cdf65aa885b533807abbcbf37006819498af95364" Oct 30 02:30:22.887208 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:22.886228 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd_odh-ai-gateway-infra(a1f42aec-2169-46f1-a837-039b920ac828)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" podUID="a1f42aec-2169-46f1-a837-039b920ac828" Oct 30 02:30:22.889825 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.889802 5998 generic.go:356] "Generic (PLEG): container finished" podID="6df3039d-5102-404f-b58f-eae1e79feb36" containerID="4f3e76c2d79457ad08d8a59ed8101754a47f4ea7154f26312d8b6fdf4eea0ef7" exitCode=60 Oct 30 02:30:22.889915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.889871 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc" event={"ID":"6df3039d-5102-404f-b58f-eae1e79feb36","Type":"ContainerDied","Data":"4f3e76c2d79457ad08d8a59ed8101754a47f4ea7154f26312d8b6fdf4eea0ef7"} Oct 30 02:30:22.891176 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.890969 5998 scope.go:122] "RemoveContainer" containerID="4f3e76c2d79457ad08d8a59ed8101754a47f4ea7154f26312d8b6fdf4eea0ef7" Oct 30 02:30:22.891463 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:22.891367 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc_odh-ai-gateway-infra(6df3039d-5102-404f-b58f-eae1e79feb36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc" podUID="6df3039d-5102-404f-b58f-eae1e79feb36" Oct 30 02:30:22.894334 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.893435 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" event={"ID":"3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb","Type":"ContainerStarted","Data":"54d2c64b5ed54555d483e99fcf6944c5bee7290278bf5c9ce33201eccc548220"} Oct 30 02:30:22.897578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.896644 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" event={"ID":"1dac6d55-be03-4ce9-b8a7-a6397747e73d","Type":"ContainerStarted","Data":"8ca010333abe3facf8128abed95933ea85cf83b7d9fbaf8a809dd12034540c18"} Oct 30 02:30:22.900005 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.899972 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" event={"ID":"512d72f9-4883-4d39-9df8-9282f56da2ac","Type":"ContainerStarted","Data":"5c5496d2d5c597896f3eb96b68ab21b73d3d452df02c45c51f049f18c365e299"} Oct 30 02:30:22.910723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.910411 5998 scope.go:122] "RemoveContainer" containerID="ef3e023f57ad7986d43f1756955746298bd01f8d4ec96b43589aa582cdb699b1" Oct 30 02:30:22.999979 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:22.999617 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" event={"ID":"e5882ea9-fe2e-4587-abb7-bb14939f53f6","Type":"ContainerStarted","Data":"8d80588fe34682f48f107a63509767d87845c5cb0b124e117fe8a098be0d579e"} Oct 30 02:30:23.024037 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.023024 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" event={"ID":"01d4f72c-dc27-4988-b301-873c276855bd","Type":"ContainerStarted","Data":"f3103f1f591bf929c011d8663f6c90fb37f46176b7d3b7bbfb8f9ba956a99290"} Oct 30 02:30:23.029399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.029363 5998 generic.go:356] "Generic (PLEG): container finished" podID="a105635a-7115-4c02-aa79-60516391579b" containerID="23fd52c0c170e63aef083e4fb54635fb4ffbdc2360a9c1fbd51931c3b3b5abfa" exitCode=60 Oct 30 02:30:23.029551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.029473 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" event={"ID":"a105635a-7115-4c02-aa79-60516391579b","Type":"ContainerDied","Data":"23fd52c0c170e63aef083e4fb54635fb4ffbdc2360a9c1fbd51931c3b3b5abfa"} Oct 30 02:30:23.030025 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.030006 5998 scope.go:122] "RemoveContainer" containerID="23fd52c0c170e63aef083e4fb54635fb4ffbdc2360a9c1fbd51931c3b3b5abfa" Oct 30 02:30:23.030479 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:23.030450 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt_odh-ai-gateway-infra(a105635a-7115-4c02-aa79-60516391579b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" podUID="a105635a-7115-4c02-aa79-60516391579b" Oct 30 02:30:23.033789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.033300 5998 scope.go:122] "RemoveContainer" containerID="1555edc8ec84f776cd24750e285b0d3a85649a39e915c06462a76670d55d26e0" Oct 30 02:30:23.039018 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.038990 5998 generic.go:356] "Generic (PLEG): container finished" podID="8e81c096-2704-4d9e-80b8-9b1f05e7151f" containerID="a46d34a6e347b327c9bd96ffb9082e2dd297f3d238ab0439465bfaeb7a10a0a8" exitCode=60 Oct 30 02:30:23.039108 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.039052 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" event={"ID":"8e81c096-2704-4d9e-80b8-9b1f05e7151f","Type":"ContainerDied","Data":"a46d34a6e347b327c9bd96ffb9082e2dd297f3d238ab0439465bfaeb7a10a0a8"} Oct 30 02:30:23.040067 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.039568 5998 scope.go:122] "RemoveContainer" containerID="a46d34a6e347b327c9bd96ffb9082e2dd297f3d238ab0439465bfaeb7a10a0a8" Oct 30 02:30:23.040067 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:23.040014 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z_odh-ai-gateway-infra(8e81c096-2704-4d9e-80b8-9b1f05e7151f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" podUID="8e81c096-2704-4d9e-80b8-9b1f05e7151f" Oct 30 02:30:23.048133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.042454 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" event={"ID":"3f69dd9b-8eda-4368-b693-33c5b9721710","Type":"ContainerStarted","Data":"9c6aecc1135c9ccc3da29f3c2cea710859113a733187cba4cdc6286468b3a836"} Oct 30 02:30:23.048133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.045992 5998 generic.go:356] "Generic (PLEG): container finished" podID="860ba768-1529-403e-ba81-13395d6cee83" containerID="eccb5b86e47fe184c5d48c9ac012c1a64c63b0ea808dddb73fff7ea558698cac" exitCode=60 Oct 30 02:30:23.048133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.046067 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" event={"ID":"860ba768-1529-403e-ba81-13395d6cee83","Type":"ContainerDied","Data":"eccb5b86e47fe184c5d48c9ac012c1a64c63b0ea808dddb73fff7ea558698cac"} Oct 30 02:30:23.048133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.046658 5998 scope.go:122] "RemoveContainer" containerID="eccb5b86e47fe184c5d48c9ac012c1a64c63b0ea808dddb73fff7ea558698cac" Oct 30 02:30:23.048133 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:23.047079 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g_odh-ai-gateway-infra(860ba768-1529-403e-ba81-13395d6cee83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" podUID="860ba768-1529-403e-ba81-13395d6cee83" Oct 30 02:30:23.048489 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.048427 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" event={"ID":"7ff76844-137e-44ec-b146-4d9c84997ada","Type":"ContainerDied","Data":"05d3493c6ec50e7c22d00f83b3523aca4959bdbe0846e76207da0faed453f61c"} Oct 30 02:30:23.048551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.048510 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp" Oct 30 02:30:23.051329 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.051215 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" event={"ID":"1d926a5c-800a-4b12-b134-be5c4ff7a619","Type":"ContainerStarted","Data":"800a41e1bc5c1a449ad452dd347231c7487b5ea1117bf2c14431053e9be5de72"} Oct 30 02:30:23.065110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.061512 5998 generic.go:356] "Generic (PLEG): container finished" podID="1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee" containerID="d73756e131cde7b384c390a639c3624c9e5055480ff0b03323e4ebe27f4039bb" exitCode=60 Oct 30 02:30:23.065110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.062811 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" event={"ID":"1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee","Type":"ContainerDied","Data":"d73756e131cde7b384c390a639c3624c9e5055480ff0b03323e4ebe27f4039bb"} Oct 30 02:30:23.076543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.076516 5998 scope.go:122] "RemoveContainer" containerID="bfd51dfd12fc0d920e83de5d303f6a3ee47c919afd18e5c105fb8e82a1d9fd9d" Oct 30 02:30:23.078948 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.078917 5998 scope.go:122] "RemoveContainer" containerID="d73756e131cde7b384c390a639c3624c9e5055480ff0b03323e4ebe27f4039bb" Oct 30 02:30:23.079461 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:23.079423 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw_odh-ai-gateway-infra(1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" podUID="1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee" Oct 30 02:30:23.117613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.117585 5998 scope.go:122] "RemoveContainer" containerID="9b3267105635039f6fce14887cdffa568d8ed5d82c9b4b35612a9f9fc490dbdb" Oct 30 02:30:23.331615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.331577 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp"] Oct 30 02:30:23.337818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.337787 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2c4b44-29888790-mqjjp"] Oct 30 02:30:23.475809 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.475772 5998 scope.go:122] "RemoveContainer" containerID="1e8cc9dcedae3a0189b91de81ab12c7c76a38fa54f0f541997685c79fda78c03" Oct 30 02:30:23.493759 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.493727 5998 scope.go:122] "RemoveContainer" containerID="86e19a0609c901c6728c0edb6c5b01b29a128bb1933e6313f086b43ac7acf357" Oct 30 02:30:23.506641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.506613 5998 scope.go:122] "RemoveContainer" containerID="a9307cc5ccab074ce26367e32ecd77e28e9306b249bb36ed62f49cdb022a85db" Oct 30 02:30:23.532910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.532362 5998 scope.go:122] "RemoveContainer" containerID="79d0c281cd74ee8e1752f6542df043055bcfeb5166ad788019792279e227aca4" Oct 30 02:30:23.532910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.532687 5998 scope.go:122] "RemoveContainer" containerID="fb366deaa1e6af17dc5d8d44b5a2453a45bd6924de38dc3a5519c08c7e33f769" Oct 30 02:30:23.936799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.936764 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn"] Oct 30 02:30:23.963914 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:23.963839 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd"] Oct 30 02:30:24.033449 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.033404 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc"] Oct 30 02:30:24.070991 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.070948 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf"] Oct 30 02:30:24.075692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.075654 5998 generic.go:356] "Generic (PLEG): container finished" podID="01d4f72c-dc27-4988-b301-873c276855bd" containerID="f3103f1f591bf929c011d8663f6c90fb37f46176b7d3b7bbfb8f9ba956a99290" exitCode=60 Oct 30 02:30:24.075863 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.075757 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" event={"ID":"01d4f72c-dc27-4988-b301-873c276855bd","Type":"ContainerDied","Data":"f3103f1f591bf929c011d8663f6c90fb37f46176b7d3b7bbfb8f9ba956a99290"} Oct 30 02:30:24.075863 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.075834 5998 scope.go:122] "RemoveContainer" containerID="26c7ffac65a2098d82f0c88a36285c9676a5074e2def1c037e1b04c0bba86ef0" Oct 30 02:30:24.076474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.076452 5998 scope.go:122] "RemoveContainer" containerID="f3103f1f591bf929c011d8663f6c90fb37f46176b7d3b7bbfb8f9ba956a99290" Oct 30 02:30:24.076939 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:24.076909 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw_odh-ai-gateway-infra(01d4f72c-dc27-4988-b301-873c276855bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" podUID="01d4f72c-dc27-4988-b301-873c276855bd" Oct 30 02:30:24.082287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.082265 5998 generic.go:356] "Generic (PLEG): container finished" podID="3f69dd9b-8eda-4368-b693-33c5b9721710" containerID="9c6aecc1135c9ccc3da29f3c2cea710859113a733187cba4cdc6286468b3a836" exitCode=60 Oct 30 02:30:24.082407 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.082336 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" event={"ID":"3f69dd9b-8eda-4368-b693-33c5b9721710","Type":"ContainerDied","Data":"9c6aecc1135c9ccc3da29f3c2cea710859113a733187cba4cdc6286468b3a836"} Oct 30 02:30:24.082805 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.082788 5998 scope.go:122] "RemoveContainer" containerID="9c6aecc1135c9ccc3da29f3c2cea710859113a733187cba4cdc6286468b3a836" Oct 30 02:30:24.083328 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:24.083298 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh_odh-ai-gateway-infra(3f69dd9b-8eda-4368-b693-33c5b9721710)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" podUID="3f69dd9b-8eda-4368-b693-33c5b9721710" Oct 30 02:30:24.088406 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.087565 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" event={"ID":"1d926a5c-800a-4b12-b134-be5c4ff7a619","Type":"ContainerStarted","Data":"42f311f1a64586e4611f77a013b44c896755cc928b6d060272f725f35dec219e"} Oct 30 02:30:24.091901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.091879 5998 generic.go:356] "Generic (PLEG): container finished" podID="046ebf9b-3d7e-4c67-bfcf-8545980b59ff" containerID="f6bc20c6b2a9cfc61514e76a54e268545b5283321c581dc3598fe5747552d2dd" exitCode=60 Oct 30 02:30:24.092012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.091960 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" event={"ID":"046ebf9b-3d7e-4c67-bfcf-8545980b59ff","Type":"ContainerDied","Data":"f6bc20c6b2a9cfc61514e76a54e268545b5283321c581dc3598fe5747552d2dd"} Oct 30 02:30:24.094844 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.094824 5998 scope.go:122] "RemoveContainer" containerID="600b823011685fdd9fe8107a0fae196342b242f75ce6eee965ef31125f978a36" Oct 30 02:30:24.098995 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.098968 5998 generic.go:356] "Generic (PLEG): container finished" podID="5bce6d68-142e-46ef-b408-89e44ca0fc10" containerID="e21556afbf5d4bc82cf3b0cc5b22a45ff7ef731dd1f7dc496a62fef9c849bd75" exitCode=60 Oct 30 02:30:24.099131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.099047 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" event={"ID":"5bce6d68-142e-46ef-b408-89e44ca0fc10","Type":"ContainerDied","Data":"e21556afbf5d4bc82cf3b0cc5b22a45ff7ef731dd1f7dc496a62fef9c849bd75"} Oct 30 02:30:24.099762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.099742 5998 scope.go:122] "RemoveContainer" containerID="e21556afbf5d4bc82cf3b0cc5b22a45ff7ef731dd1f7dc496a62fef9c849bd75" Oct 30 02:30:24.115159 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:24.100248 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k_odh-ai-gateway-infra(5bce6d68-142e-46ef-b408-89e44ca0fc10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" podUID="5bce6d68-142e-46ef-b408-89e44ca0fc10" Oct 30 02:30:24.139888 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.139846 5998 generic.go:356] "Generic (PLEG): container finished" podID="512be8ad-51e7-4919-a456-9fcfafab46bd" containerID="57ff8ac410b8862c3777ce22287520b12cc7f9427839a122162ddab986736a27" exitCode=60 Oct 30 02:30:24.140122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.139969 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" event={"ID":"512be8ad-51e7-4919-a456-9fcfafab46bd","Type":"ContainerDied","Data":"57ff8ac410b8862c3777ce22287520b12cc7f9427839a122162ddab986736a27"} Oct 30 02:30:24.140538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.140503 5998 scope.go:122] "RemoveContainer" containerID="57ff8ac410b8862c3777ce22287520b12cc7f9427839a122162ddab986736a27" Oct 30 02:30:24.141431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.141368 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn"] Oct 30 02:30:24.141746 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:24.141709 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq_odh-ai-gateway-infra(512be8ad-51e7-4919-a456-9fcfafab46bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" podUID="512be8ad-51e7-4919-a456-9fcfafab46bd" Oct 30 02:30:24.151704 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.150605 5998 scope.go:122] "RemoveContainer" containerID="06ba2ce69cb9eb1821fef5fdb771d2512a2d3740b31562f876c764ccc8cec9bc" Oct 30 02:30:24.155612 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.155570 5998 generic.go:356] "Generic (PLEG): container finished" podID="3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb" containerID="54d2c64b5ed54555d483e99fcf6944c5bee7290278bf5c9ce33201eccc548220" exitCode=60 Oct 30 02:30:24.155771 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.155650 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" event={"ID":"3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb","Type":"ContainerDied","Data":"54d2c64b5ed54555d483e99fcf6944c5bee7290278bf5c9ce33201eccc548220"} Oct 30 02:30:24.161262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.158536 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" event={"ID":"1dac6d55-be03-4ce9-b8a7-a6397747e73d","Type":"ContainerStarted","Data":"3d9086dab4379432ea308ad7b591c989fb1b3b1da907058abd6c69aefd966c58"} Oct 30 02:30:24.175912 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.169403 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr"] Oct 30 02:30:24.196396 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.196360 5998 scope.go:122] "RemoveContainer" containerID="edb54ca7e39dadc76acc72f712f3eff40a4e350b7ab8a7cde2b6cfa9b6634ee2" Oct 30 02:30:24.196648 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.196618 5998 generic.go:356] "Generic (PLEG): container finished" podID="512d72f9-4883-4d39-9df8-9282f56da2ac" containerID="5c5496d2d5c597896f3eb96b68ab21b73d3d452df02c45c51f049f18c365e299" exitCode=60 Oct 30 02:30:24.196810 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.196788 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" event={"ID":"512d72f9-4883-4d39-9df8-9282f56da2ac","Type":"ContainerDied","Data":"5c5496d2d5c597896f3eb96b68ab21b73d3d452df02c45c51f049f18c365e299"} Oct 30 02:30:24.242028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.240390 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:24.247812 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.244288 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:24.247812 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.246275 5998 patch_prober.go:29] interesting pod/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 30 02:30:24.247812 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.246331 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" podUID="1d926a5c-800a-4b12-b134-be5c4ff7a619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 30 02:30:24.254031 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.253971 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:24.283108 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.279157 5998 generic.go:356] "Generic (PLEG): container finished" podID="e5882ea9-fe2e-4587-abb7-bb14939f53f6" containerID="8d80588fe34682f48f107a63509767d87845c5cb0b124e117fe8a098be0d579e" exitCode=60 Oct 30 02:30:24.283108 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.280417 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" event={"ID":"e5882ea9-fe2e-4587-abb7-bb14939f53f6","Type":"ContainerDied","Data":"8d80588fe34682f48f107a63509767d87845c5cb0b124e117fe8a098be0d579e"} Oct 30 02:30:24.283108 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.280885 5998 scope.go:122] "RemoveContainer" containerID="8d80588fe34682f48f107a63509767d87845c5cb0b124e117fe8a098be0d579e" Oct 30 02:30:24.283108 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:24.281323 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6_odh-ai-gateway-infra(e5882ea9-fe2e-4587-abb7-bb14939f53f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" podUID="e5882ea9-fe2e-4587-abb7-bb14939f53f6" Oct 30 02:30:24.386358 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.386321 5998 scope.go:122] "RemoveContainer" containerID="27c4826c0628cf90c7f034eec6af68694bbe65e42077d4f46bf99fa840712636" Oct 30 02:30:24.397769 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.397744 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt"] Oct 30 02:30:24.454243 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.454204 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z"] Oct 30 02:30:24.466512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.464596 5998 scope.go:122] "RemoveContainer" containerID="8ef843f3d22e1c1029643acdb1e32dfea707800156c7a4cd6d1b284c6c5b5637" Oct 30 02:30:24.476446 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.476407 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw"] Oct 30 02:30:24.493676 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.493259 5998 scope.go:122] "RemoveContainer" containerID="b947ecaf282ade6974160eab1ab8046a36d627d04da067735804d90d1d0613ef" Oct 30 02:30:24.541769 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.541673 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" podStartSLOduration=3.541653475 podStartE2EDuration="3.541653475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:24.374430015 +0000 UTC m=+3653.953221099" watchObservedRunningTime="2026-10-30 02:30:24.541653475 +0000 UTC m=+3654.120444558" Oct 30 02:30:24.543984 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.543930 5998 scope.go:122] "RemoveContainer" containerID="55cba0eb40314867cf6b2d66f160469b841bb0c32284f6987eafd0778727b966" Oct 30 02:30:24.545321 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.545270 5998 scope.go:122] "RemoveContainer" containerID="bfe24a95fde280b39bfc982bd812c048dbe52ef1d7489db9b9f45ea694d55eb1" Oct 30 02:30:24.545708 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.545636 5998 scope.go:122] "RemoveContainer" containerID="e3cd7f2ab448a9fb591dcb3c307bc71ea1ee8373a79b61aa5a92080b6704e74e" Oct 30 02:30:24.553205 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.552431 5998 scope.go:122] "RemoveContainer" containerID="4808839e8ecf47d51616536ac0709d0b7dff0daac511e5a59c6ce6f9dbe81f12" Oct 30 02:30:24.554225 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.554202 5998 scope.go:122] "RemoveContainer" containerID="beeb1625a64a57cc79961b5b7fa1df2ceae6b5c926f85462418aa49c50a11b81" Oct 30 02:30:24.703188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.702856 5998 scope.go:122] "RemoveContainer" containerID="1e12b2bcf54976d519b107ab2a34702c508caeb6f5ebc61b072d92a53e171474" Oct 30 02:30:24.779798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.779751 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:24.787040 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.787014 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" Oct 30 02:30:24.788137 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.788112 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff76844-137e-44ec-b146-4d9c84997ada" path="/var/lib/kubelet/pods/7ff76844-137e-44ec-b146-4d9c84997ada/volumes" Oct 30 02:30:24.788991 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.788971 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g"] Oct 30 02:30:24.789190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.789175 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw"] Oct 30 02:30:24.789313 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.789302 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh"] Oct 30 02:30:24.845397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:24.845286 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6"] Oct 30 02:30:25.032519 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.021341 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/046ebf9b-3d7e-4c67-bfcf-8545980b59ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046ebf9b-3d7e-4c67-bfcf-8545980b59ff-maas-api-service-ca\") pod \"046ebf9b-3d7e-4c67-bfcf-8545980b59ff\" (UID: \"046ebf9b-3d7e-4c67-bfcf-8545980b59ff\") " Oct 30 02:30:25.032519 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.021495 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/046ebf9b-3d7e-4c67-bfcf-8545980b59ff-kube-api-access-7vns4\" (UniqueName: \"kubernetes.io/projected/046ebf9b-3d7e-4c67-bfcf-8545980b59ff-kube-api-access-7vns4\") pod \"046ebf9b-3d7e-4c67-bfcf-8545980b59ff\" (UID: \"046ebf9b-3d7e-4c67-bfcf-8545980b59ff\") " Oct 30 02:30:25.068113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.065342 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046ebf9b-3d7e-4c67-bfcf-8545980b59ff-maas-api-service-ca" pod "046ebf9b-3d7e-4c67-bfcf-8545980b59ff" (UID: "046ebf9b-3d7e-4c67-bfcf-8545980b59ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:25.126029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.097160 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" podStartSLOduration=5.097138384 podStartE2EDuration="5.097138384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:30:24.779549632 +0000 UTC m=+3654.358340717" watchObservedRunningTime="2026-10-30 02:30:25.097138384 +0000 UTC m=+3654.675929467" Oct 30 02:30:25.142814 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:25.142765 5998 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96bc88e1_7b35_4b54_b12f_69227ae41a88.slice/crio-conmon-0bae572f93f41aff92d8c004345ded2c236edc1b4fe05fa96089f74fd2c92cfd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0771e950_74b9_45cb_a7be_ccb30c8c8c65.slice/crio-conmon-e7863ad2ff142bff058f26383400d5c2e71aa4c832457ba9f0e0bbb88a612d4b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 02:30:25.154013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.153968 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/046ebf9b-3d7e-4c67-bfcf-8545980b59ff-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:25.195286 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.187661 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046ebf9b-3d7e-4c67-bfcf-8545980b59ff-kube-api-access-7vns4" pod "046ebf9b-3d7e-4c67-bfcf-8545980b59ff" (UID: "046ebf9b-3d7e-4c67-bfcf-8545980b59ff"). InnerVolumeSpecName "kube-api-access-7vns4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:25.287364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.283500 5998 patch_prober.go:29] interesting pod/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 30 02:30:25.287364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.283554 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" podUID="1d926a5c-800a-4b12-b134-be5c4ff7a619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 30 02:30:25.366745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.361765 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vns4\" (UniqueName: \"kubernetes.io/projected/046ebf9b-3d7e-4c67-bfcf-8545980b59ff-kube-api-access-7vns4\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:25.468320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.468284 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k"] Oct 30 02:30:25.495580 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.495534 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" Oct 30 02:30:25.498473 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.498427 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc" Oct 30 02:30:25.500048 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.500010 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" Oct 30 02:30:25.564394 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.564364 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" Oct 30 02:30:25.567405 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.567374 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc" event={"ID":"6df3039d-5102-404f-b58f-eae1e79feb36","Type":"ContainerDied","Data":"ba4d93314e7e727725b05fc44ad5dcf5fece5703ac4cd47f155b71d2f7175a37"} Oct 30 02:30:25.567607 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.567589 5998 scope.go:122] "RemoveContainer" containerID="4f3e76c2d79457ad08d8a59ed8101754a47f4ea7154f26312d8b6fdf4eea0ef7" Oct 30 02:30:25.567759 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.567395 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc" Oct 30 02:30:25.569592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.569566 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" event={"ID":"3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb","Type":"ContainerDied","Data":"47b5fd8c5fc05c932ac941f4eb4181978e34ccfe0a21634be509345be2ba6217"} Oct 30 02:30:25.570135 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.569628 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn" Oct 30 02:30:25.573064 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.573038 5998 generic.go:356] "Generic (PLEG): container finished" podID="0771e950-74b9-45cb-a7be-ccb30c8c8c65" containerID="e7863ad2ff142bff058f26383400d5c2e71aa4c832457ba9f0e0bbb88a612d4b" exitCode=60 Oct 30 02:30:25.573310 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.573118 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" event={"ID":"0771e950-74b9-45cb-a7be-ccb30c8c8c65","Type":"ContainerDied","Data":"e7863ad2ff142bff058f26383400d5c2e71aa4c832457ba9f0e0bbb88a612d4b"} Oct 30 02:30:25.573740 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.573720 5998 scope.go:122] "RemoveContainer" containerID="e7863ad2ff142bff058f26383400d5c2e71aa4c832457ba9f0e0bbb88a612d4b" Oct 30 02:30:25.574151 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:25.574070 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf_odh-ai-gateway-infra(0771e950-74b9-45cb-a7be-ccb30c8c8c65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" podUID="0771e950-74b9-45cb-a7be-ccb30c8c8c65" Oct 30 02:30:25.589788 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.589685 5998 scope.go:122] "RemoveContainer" containerID="54d2c64b5ed54555d483e99fcf6944c5bee7290278bf5c9ce33201eccc548220" Oct 30 02:30:25.597475 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.595037 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" Oct 30 02:30:25.597760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.597631 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn" event={"ID":"046ebf9b-3d7e-4c67-bfcf-8545980b59ff","Type":"ContainerDied","Data":"da8569ee276fbe190f7064e8444c12bb71d787d04c815d2dce451fb744fc932f"} Oct 30 02:30:25.597760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.597694 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq"] Oct 30 02:30:25.639081 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.639045 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" event={"ID":"be309ab3-73d7-42c0-92a8-0fc686504795","Type":"ContainerDied","Data":"97b78f0329c3df4253bb8d2403a4bdb3f979fc455367f30dd74d31454f7ecb82"} Oct 30 02:30:25.639320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.639231 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf" Oct 30 02:30:25.641403 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.641139 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" event={"ID":"a1f42aec-2169-46f1-a837-039b920ac828","Type":"ContainerDied","Data":"c4e32547cc1aa242ae6cfd751a46f855ce64edd28f37a1c16448214085511d8e"} Oct 30 02:30:25.641403 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.641209 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd" Oct 30 02:30:25.662200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.652466 5998 generic.go:356] "Generic (PLEG): container finished" podID="96bc88e1-7b35-4b54-b12f-69227ae41a88" containerID="0bae572f93f41aff92d8c004345ded2c236edc1b4fe05fa96089f74fd2c92cfd" exitCode=60 Oct 30 02:30:25.662200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.658698 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" event={"ID":"96bc88e1-7b35-4b54-b12f-69227ae41a88","Type":"ContainerDied","Data":"0bae572f93f41aff92d8c004345ded2c236edc1b4fe05fa96089f74fd2c92cfd"} Oct 30 02:30:25.662200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.658774 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:25.662200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.659292 5998 scope.go:122] "RemoveContainer" containerID="0bae572f93f41aff92d8c004345ded2c236edc1b4fe05fa96089f74fd2c92cfd" Oct 30 02:30:25.662200 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:25.659804 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj_odh-ai-gateway-infra(96bc88e1-7b35-4b54-b12f-69227ae41a88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" podUID="96bc88e1-7b35-4b54-b12f-69227ae41a88" Oct 30 02:30:25.667468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.666257 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df3039d-5102-404f-b58f-eae1e79feb36-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df3039d-5102-404f-b58f-eae1e79feb36-maas-api-service-ca\") pod \"6df3039d-5102-404f-b58f-eae1e79feb36\" (UID: \"6df3039d-5102-404f-b58f-eae1e79feb36\") " Oct 30 02:30:25.667468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.666298 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1f42aec-2169-46f1-a837-039b920ac828-kube-api-access-svhbs\" (UniqueName: \"kubernetes.io/projected/a1f42aec-2169-46f1-a837-039b920ac828-kube-api-access-svhbs\") pod \"a1f42aec-2169-46f1-a837-039b920ac828\" (UID: \"a1f42aec-2169-46f1-a837-039b920ac828\") " Oct 30 02:30:25.667468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.666424 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df3039d-5102-404f-b58f-eae1e79feb36-kube-api-access-ph497\" (UniqueName: \"kubernetes.io/projected/6df3039d-5102-404f-b58f-eae1e79feb36-kube-api-access-ph497\") pod \"6df3039d-5102-404f-b58f-eae1e79feb36\" (UID: \"6df3039d-5102-404f-b58f-eae1e79feb36\") " Oct 30 02:30:25.667468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.666431 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:30:25.667468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.666469 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be309ab3-73d7-42c0-92a8-0fc686504795-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be309ab3-73d7-42c0-92a8-0fc686504795-maas-api-service-ca\") pod \"be309ab3-73d7-42c0-92a8-0fc686504795\" (UID: \"be309ab3-73d7-42c0-92a8-0fc686504795\") " Oct 30 02:30:25.667468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.666517 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb-kube-api-access-wvcsj\" (UniqueName: \"kubernetes.io/projected/3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb-kube-api-access-wvcsj\") pod \"3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb\" (UID: \"3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb\") " Oct 30 02:30:25.667468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.666547 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1f42aec-2169-46f1-a837-039b920ac828-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f42aec-2169-46f1-a837-039b920ac828-maas-api-service-ca\") pod \"a1f42aec-2169-46f1-a837-039b920ac828\" (UID: \"a1f42aec-2169-46f1-a837-039b920ac828\") " Oct 30 02:30:25.667468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.666580 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb-maas-api-service-ca\") pod \"3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb\" (UID: \"3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb\") " Oct 30 02:30:25.667468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.666601 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be309ab3-73d7-42c0-92a8-0fc686504795-kube-api-access-77n7p\" (UniqueName: \"kubernetes.io/projected/be309ab3-73d7-42c0-92a8-0fc686504795-kube-api-access-77n7p\") pod \"be309ab3-73d7-42c0-92a8-0fc686504795\" (UID: \"be309ab3-73d7-42c0-92a8-0fc686504795\") " Oct 30 02:30:25.669326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.668992 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f42aec-2169-46f1-a837-039b920ac828-maas-api-service-ca" pod "a1f42aec-2169-46f1-a837-039b920ac828" (UID: "a1f42aec-2169-46f1-a837-039b920ac828"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:25.669854 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.669817 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb-maas-api-service-ca" pod "3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb" (UID: "3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:25.670877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.670850 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be309ab3-73d7-42c0-92a8-0fc686504795-maas-api-service-ca" pod "be309ab3-73d7-42c0-92a8-0fc686504795" (UID: "be309ab3-73d7-42c0-92a8-0fc686504795"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:25.674506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.674479 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df3039d-5102-404f-b58f-eae1e79feb36-maas-api-service-ca" pod "6df3039d-5102-404f-b58f-eae1e79feb36" (UID: "6df3039d-5102-404f-b58f-eae1e79feb36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:25.754121 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:25.754056 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:30:25.770941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.770708 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6df3039d-5102-404f-b58f-eae1e79feb36-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:25.770941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.770745 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be309ab3-73d7-42c0-92a8-0fc686504795-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:25.770941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.770758 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1f42aec-2169-46f1-a837-039b920ac828-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:25.770941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.770774 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:25.792746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.792713 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn"] Oct 30 02:30:25.799905 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.799832 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4593af-29888790-9wkhn"] Oct 30 02:30:25.831184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.830511 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df3039d-5102-404f-b58f-eae1e79feb36-kube-api-access-ph497" pod "6df3039d-5102-404f-b58f-eae1e79feb36" (UID: "6df3039d-5102-404f-b58f-eae1e79feb36"). InnerVolumeSpecName "kube-api-access-ph497". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:25.831184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.830897 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb-kube-api-access-wvcsj" pod "3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb" (UID: "3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb"). InnerVolumeSpecName "kube-api-access-wvcsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:25.846247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.838457 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f42aec-2169-46f1-a837-039b920ac828-kube-api-access-svhbs" pod "a1f42aec-2169-46f1-a837-039b920ac828" (UID: "a1f42aec-2169-46f1-a837-039b920ac828"). InnerVolumeSpecName "kube-api-access-svhbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:25.846247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.838521 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be309ab3-73d7-42c0-92a8-0fc686504795-kube-api-access-77n7p" pod "be309ab3-73d7-42c0-92a8-0fc686504795" (UID: "be309ab3-73d7-42c0-92a8-0fc686504795"). InnerVolumeSpecName "kube-api-access-77n7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:25.882069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.879894 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" Oct 30 02:30:25.954940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.942400 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a105635a-7115-4c02-aa79-60516391579b-kube-api-access-pxtdv\" (UniqueName: \"kubernetes.io/projected/a105635a-7115-4c02-aa79-60516391579b-kube-api-access-pxtdv\") pod \"a105635a-7115-4c02-aa79-60516391579b\" (UID: \"a105635a-7115-4c02-aa79-60516391579b\") " Oct 30 02:30:25.954940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.942703 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a105635a-7115-4c02-aa79-60516391579b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a105635a-7115-4c02-aa79-60516391579b-maas-api-service-ca\") pod \"a105635a-7115-4c02-aa79-60516391579b\" (UID: \"a105635a-7115-4c02-aa79-60516391579b\") " Oct 30 02:30:25.966216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.966176 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a105635a-7115-4c02-aa79-60516391579b-kube-api-access-pxtdv" pod "a105635a-7115-4c02-aa79-60516391579b" (UID: "a105635a-7115-4c02-aa79-60516391579b"). InnerVolumeSpecName "kube-api-access-pxtdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:25.979849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:25.979818 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" Oct 30 02:30:26.038697 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.038644 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a105635a-7115-4c02-aa79-60516391579b-maas-api-service-ca" pod "a105635a-7115-4c02-aa79-60516391579b" (UID: "a105635a-7115-4c02-aa79-60516391579b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:26.039475 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.039145 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svhbs\" (UniqueName: \"kubernetes.io/projected/a1f42aec-2169-46f1-a837-039b920ac828-kube-api-access-svhbs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.039475 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.039264 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxtdv\" (UniqueName: \"kubernetes.io/projected/a105635a-7115-4c02-aa79-60516391579b-kube-api-access-pxtdv\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.039475 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.039285 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph497\" (UniqueName: \"kubernetes.io/projected/6df3039d-5102-404f-b58f-eae1e79feb36-kube-api-access-ph497\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.039475 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.039312 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvcsj\" (UniqueName: \"kubernetes.io/projected/3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb-kube-api-access-wvcsj\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.039475 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.039328 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77n7p\" (UniqueName: \"kubernetes.io/projected/be309ab3-73d7-42c0-92a8-0fc686504795-kube-api-access-77n7p\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.050391 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.050232 5998 scope.go:122] "RemoveContainer" containerID="79d0c281cd74ee8e1752f6542df043055bcfeb5166ad788019792279e227aca4" Oct 30 02:30:26.071491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.071462 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" Oct 30 02:30:26.161883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.158954 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e81c096-2704-4d9e-80b8-9b1f05e7151f-kube-api-access-m8qj6\" (UniqueName: \"kubernetes.io/projected/8e81c096-2704-4d9e-80b8-9b1f05e7151f-kube-api-access-m8qj6\") pod \"8e81c096-2704-4d9e-80b8-9b1f05e7151f\" (UID: \"8e81c096-2704-4d9e-80b8-9b1f05e7151f\") " Oct 30 02:30:26.161883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.159012 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee-kube-api-access-wg6kp\" (UniqueName: \"kubernetes.io/projected/1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee-kube-api-access-wg6kp\") pod \"1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee\" (UID: \"1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee\") " Oct 30 02:30:26.161883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.159066 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e81c096-2704-4d9e-80b8-9b1f05e7151f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e81c096-2704-4d9e-80b8-9b1f05e7151f-maas-api-service-ca\") pod \"8e81c096-2704-4d9e-80b8-9b1f05e7151f\" (UID: \"8e81c096-2704-4d9e-80b8-9b1f05e7151f\") " Oct 30 02:30:26.161883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.159428 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee-maas-api-service-ca\") pod \"1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee\" (UID: \"1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee\") " Oct 30 02:30:26.162991 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.162749 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a105635a-7115-4c02-aa79-60516391579b-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.163496 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.163277 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee-maas-api-service-ca" pod "1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee" (UID: "1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:26.163774 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.163717 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e81c096-2704-4d9e-80b8-9b1f05e7151f-maas-api-service-ca" pod "8e81c096-2704-4d9e-80b8-9b1f05e7151f" (UID: "8e81c096-2704-4d9e-80b8-9b1f05e7151f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:26.165679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.164605 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee-kube-api-access-wg6kp" pod "1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee" (UID: "1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee"). InnerVolumeSpecName "kube-api-access-wg6kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:26.166880 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.166853 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e81c096-2704-4d9e-80b8-9b1f05e7151f-kube-api-access-m8qj6" pod "8e81c096-2704-4d9e-80b8-9b1f05e7151f" (UID: "8e81c096-2704-4d9e-80b8-9b1f05e7151f"). InnerVolumeSpecName "kube-api-access-m8qj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:26.264728 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.264685 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8qj6\" (UniqueName: \"kubernetes.io/projected/8e81c096-2704-4d9e-80b8-9b1f05e7151f-kube-api-access-m8qj6\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.264917 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.264733 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg6kp\" (UniqueName: \"kubernetes.io/projected/1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee-kube-api-access-wg6kp\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.264917 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.264749 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e81c096-2704-4d9e-80b8-9b1f05e7151f-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.264917 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.264767 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.268483 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.268447 5998 patch_prober.go:29] interesting pod/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 30 02:30:26.268617 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.268505 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" podUID="1d926a5c-800a-4b12-b134-be5c4ff7a619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 30 02:30:26.293895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.278521 5998 scope.go:122] "RemoveContainer" containerID="f6bc20c6b2a9cfc61514e76a54e268545b5283321c581dc3598fe5747552d2dd" Oct 30 02:30:26.293895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.292268 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" Oct 30 02:30:26.386699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.367243 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/512d72f9-4883-4d39-9df8-9282f56da2ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512d72f9-4883-4d39-9df8-9282f56da2ac-maas-api-service-ca\") pod \"512d72f9-4883-4d39-9df8-9282f56da2ac\" (UID: \"512d72f9-4883-4d39-9df8-9282f56da2ac\") " Oct 30 02:30:26.386699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.367570 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/512d72f9-4883-4d39-9df8-9282f56da2ac-kube-api-access-wq6kr\" (UniqueName: \"kubernetes.io/projected/512d72f9-4883-4d39-9df8-9282f56da2ac-kube-api-access-wq6kr\") pod \"512d72f9-4883-4d39-9df8-9282f56da2ac\" (UID: \"512d72f9-4883-4d39-9df8-9282f56da2ac\") " Oct 30 02:30:26.386699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.375456 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512d72f9-4883-4d39-9df8-9282f56da2ac-maas-api-service-ca" pod "512d72f9-4883-4d39-9df8-9282f56da2ac" (UID: "512d72f9-4883-4d39-9df8-9282f56da2ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:26.386699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.375696 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512d72f9-4883-4d39-9df8-9282f56da2ac-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.455424 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.407769 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:26.552252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.552214 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" Oct 30 02:30:26.554773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.554745 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" Oct 30 02:30:26.556513 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.556145 5998 scope.go:122] "RemoveContainer" containerID="17635562922cb55f8bfa8158240dfd3128372b51d77d25d3bcf6b7c573960e14" Oct 30 02:30:26.557479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.557450 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512d72f9-4883-4d39-9df8-9282f56da2ac-kube-api-access-wq6kr" pod "512d72f9-4883-4d39-9df8-9282f56da2ac" (UID: "512d72f9-4883-4d39-9df8-9282f56da2ac"). InnerVolumeSpecName "kube-api-access-wq6kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:26.564642 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.564593 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" Oct 30 02:30:26.575405 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.575142 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="046ebf9b-3d7e-4c67-bfcf-8545980b59ff" path="/var/lib/kubelet/pods/046ebf9b-3d7e-4c67-bfcf-8545980b59ff/volumes" Oct 30 02:30:26.580311 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.580283 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860ba768-1529-403e-ba81-13395d6cee83-kube-api-access-n4lln\" (UniqueName: \"kubernetes.io/projected/860ba768-1529-403e-ba81-13395d6cee83-kube-api-access-n4lln\") pod \"860ba768-1529-403e-ba81-13395d6cee83\" (UID: \"860ba768-1529-403e-ba81-13395d6cee83\") " Oct 30 02:30:26.581207 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.581186 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5882ea9-fe2e-4587-abb7-bb14939f53f6-kube-api-access-jn4s7\" (UniqueName: \"kubernetes.io/projected/e5882ea9-fe2e-4587-abb7-bb14939f53f6-kube-api-access-jn4s7\") pod \"e5882ea9-fe2e-4587-abb7-bb14939f53f6\" (UID: \"e5882ea9-fe2e-4587-abb7-bb14939f53f6\") " Oct 30 02:30:26.582832 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.582788 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f69dd9b-8eda-4368-b693-33c5b9721710-maas-api-service-ca" pod "3f69dd9b-8eda-4368-b693-33c5b9721710" (UID: "3f69dd9b-8eda-4368-b693-33c5b9721710"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:26.583011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.582991 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f69dd9b-8eda-4368-b693-33c5b9721710-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f69dd9b-8eda-4368-b693-33c5b9721710-maas-api-service-ca\") pod \"3f69dd9b-8eda-4368-b693-33c5b9721710\" (UID: \"3f69dd9b-8eda-4368-b693-33c5b9721710\") " Oct 30 02:30:26.583248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.583232 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/860ba768-1529-403e-ba81-13395d6cee83-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/860ba768-1529-403e-ba81-13395d6cee83-maas-api-service-ca\") pod \"860ba768-1529-403e-ba81-13395d6cee83\" (UID: \"860ba768-1529-403e-ba81-13395d6cee83\") " Oct 30 02:30:26.583786 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.583670 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860ba768-1529-403e-ba81-13395d6cee83-maas-api-service-ca" pod "860ba768-1529-403e-ba81-13395d6cee83" (UID: "860ba768-1529-403e-ba81-13395d6cee83"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:26.583946 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.583929 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f69dd9b-8eda-4368-b693-33c5b9721710-kube-api-access-hh5lf\" (UniqueName: \"kubernetes.io/projected/3f69dd9b-8eda-4368-b693-33c5b9721710-kube-api-access-hh5lf\") pod \"3f69dd9b-8eda-4368-b693-33c5b9721710\" (UID: \"3f69dd9b-8eda-4368-b693-33c5b9721710\") " Oct 30 02:30:26.584582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.584561 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5882ea9-fe2e-4587-abb7-bb14939f53f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5882ea9-fe2e-4587-abb7-bb14939f53f6-maas-api-service-ca\") pod \"e5882ea9-fe2e-4587-abb7-bb14939f53f6\" (UID: \"e5882ea9-fe2e-4587-abb7-bb14939f53f6\") " Oct 30 02:30:26.589947 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.586016 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5882ea9-fe2e-4587-abb7-bb14939f53f6-maas-api-service-ca" pod "e5882ea9-fe2e-4587-abb7-bb14939f53f6" (UID: "e5882ea9-fe2e-4587-abb7-bb14939f53f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:26.591223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.591202 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f69dd9b-8eda-4368-b693-33c5b9721710-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.591355 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.591341 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/860ba768-1529-403e-ba81-13395d6cee83-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.591435 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.591426 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq6kr\" (UniqueName: \"kubernetes.io/projected/512d72f9-4883-4d39-9df8-9282f56da2ac-kube-api-access-wq6kr\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.655742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.654959 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" Oct 30 02:30:26.669991 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.669142 5998 scope.go:122] "RemoveContainer" containerID="a60df0e16ddcec6e2600fd4cdf65aa885b533807abbcbf37006819498af95364" Oct 30 02:30:26.683060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.682329 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5882ea9-fe2e-4587-abb7-bb14939f53f6-kube-api-access-jn4s7" pod "e5882ea9-fe2e-4587-abb7-bb14939f53f6" (UID: "e5882ea9-fe2e-4587-abb7-bb14939f53f6"). InnerVolumeSpecName "kube-api-access-jn4s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:26.750217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.693542 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bce6d68-142e-46ef-b408-89e44ca0fc10-kube-api-access-gbr5n\" (UniqueName: \"kubernetes.io/projected/5bce6d68-142e-46ef-b408-89e44ca0fc10-kube-api-access-gbr5n\") pod \"5bce6d68-142e-46ef-b408-89e44ca0fc10\" (UID: \"5bce6d68-142e-46ef-b408-89e44ca0fc10\") " Oct 30 02:30:26.750217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.693684 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bce6d68-142e-46ef-b408-89e44ca0fc10-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bce6d68-142e-46ef-b408-89e44ca0fc10-maas-api-service-ca\") pod \"5bce6d68-142e-46ef-b408-89e44ca0fc10\" (UID: \"5bce6d68-142e-46ef-b408-89e44ca0fc10\") " Oct 30 02:30:26.750217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.694067 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bce6d68-142e-46ef-b408-89e44ca0fc10-maas-api-service-ca" pod "5bce6d68-142e-46ef-b408-89e44ca0fc10" (UID: "5bce6d68-142e-46ef-b408-89e44ca0fc10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:26.750217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.695718 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn4s7\" (UniqueName: \"kubernetes.io/projected/e5882ea9-fe2e-4587-abb7-bb14939f53f6-kube-api-access-jn4s7\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.750217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.695742 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bce6d68-142e-46ef-b408-89e44ca0fc10-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.750217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.695757 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5882ea9-fe2e-4587-abb7-bb14939f53f6-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.754980 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.754877 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860ba768-1529-403e-ba81-13395d6cee83-kube-api-access-n4lln" pod "860ba768-1529-403e-ba81-13395d6cee83" (UID: "860ba768-1529-403e-ba81-13395d6cee83"). InnerVolumeSpecName "kube-api-access-n4lln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:26.761384 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.759697 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f69dd9b-8eda-4368-b693-33c5b9721710-kube-api-access-hh5lf" pod "3f69dd9b-8eda-4368-b693-33c5b9721710" (UID: "3f69dd9b-8eda-4368-b693-33c5b9721710"). InnerVolumeSpecName "kube-api-access-hh5lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:26.766308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.765263 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" event={"ID":"5bce6d68-142e-46ef-b408-89e44ca0fc10","Type":"ContainerDied","Data":"ccafd81f214e1efcef9fb02f4d99efd01629930a1bfcfac3af939333eaa3b5c5"} Oct 30 02:30:26.766308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.765385 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k" Oct 30 02:30:26.766832 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.766805 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bce6d68-142e-46ef-b408-89e44ca0fc10-kube-api-access-gbr5n" pod "5bce6d68-142e-46ef-b408-89e44ca0fc10" (UID: "5bce6d68-142e-46ef-b408-89e44ca0fc10"). InnerVolumeSpecName "kube-api-access-gbr5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:26.767706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.767638 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" Oct 30 02:30:26.768568 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.768544 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" event={"ID":"512be8ad-51e7-4919-a456-9fcfafab46bd","Type":"ContainerDied","Data":"be7b281637325fb5da2e13a7d3d544162ff987003df39f297f375018cbc9832e"} Oct 30 02:30:26.771571 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.771481 5998 generic.go:356] "Generic (PLEG): container finished" podID="2f3ead60-88ca-42af-9e25-7822589058e4" containerID="6c7f23c8f09d2384f01c1c24c8e3adaeed5293f578e083da2ede26f06f312787" exitCode=60 Oct 30 02:30:26.771944 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.771699 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" event={"ID":"2f3ead60-88ca-42af-9e25-7822589058e4","Type":"ContainerDied","Data":"6c7f23c8f09d2384f01c1c24c8e3adaeed5293f578e083da2ede26f06f312787"} Oct 30 02:30:26.773049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.772504 5998 scope.go:122] "RemoveContainer" containerID="6c7f23c8f09d2384f01c1c24c8e3adaeed5293f578e083da2ede26f06f312787" Oct 30 02:30:26.773724 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:26.773558 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs_odh-ai-gateway-infra(2f3ead60-88ca-42af-9e25-7822589058e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" podUID="2f3ead60-88ca-42af-9e25-7822589058e4" Oct 30 02:30:26.779812 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.779707 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf"] Oct 30 02:30:26.783795 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.783763 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" Oct 30 02:30:26.784590 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.784302 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr" event={"ID":"512d72f9-4883-4d39-9df8-9282f56da2ac","Type":"ContainerDied","Data":"55656aeb1553dcae558c0ccf770bd981151adc537fb4ff2a1a68935b03e1cddf"} Oct 30 02:30:26.786160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.785818 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c75203-29888790-qdlcf"] Oct 30 02:30:26.787790 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.787476 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" event={"ID":"9e4909d0-f3ae-423a-a478-4ee5ce9734a1","Type":"ContainerStarted","Data":"7e96b738e05652e8598fca8c9e6f4393fd7c9551cb8e37976c0ea08a6843a528"} Oct 30 02:30:26.788431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.788409 5998 scope.go:122] "RemoveContainer" containerID="7e96b738e05652e8598fca8c9e6f4393fd7c9551cb8e37976c0ea08a6843a528" Oct 30 02:30:26.788821 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:26.788797 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc_odh-ai-gateway-infra(9e4909d0-f3ae-423a-a478-4ee5ce9734a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" podUID="9e4909d0-f3ae-423a-a478-4ee5ce9734a1" Oct 30 02:30:26.790598 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.789927 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" event={"ID":"e5882ea9-fe2e-4587-abb7-bb14939f53f6","Type":"ContainerDied","Data":"3d2b005d212b83bbbef2d13ea9cab2f86525bf09a8d277c28ef63d9b6f7f39f2"} Oct 30 02:30:26.790734 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.789937 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6" Oct 30 02:30:26.794952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.793841 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" event={"ID":"daaa2b88-a49a-4a88-8325-33307e969074","Type":"ContainerStarted","Data":"0f64758a273c7816f2df6c0f9c08c7a5aa2ca0d4e0b784affbab53611ad05ff9"} Oct 30 02:30:26.795637 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.795612 5998 scope.go:122] "RemoveContainer" containerID="0f64758a273c7816f2df6c0f9c08c7a5aa2ca0d4e0b784affbab53611ad05ff9" Oct 30 02:30:26.795959 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:26.795934 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2_odh-ai-gateway-infra(daaa2b88-a49a-4a88-8325-33307e969074)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" podUID="daaa2b88-a49a-4a88-8325-33307e969074" Oct 30 02:30:26.796196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.796160 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" event={"ID":"01d4f72c-dc27-4988-b301-873c276855bd","Type":"ContainerDied","Data":"c514c46fd577179a57c3681a35c5ef5aab6d148e58fae47ab874381283144a61"} Oct 30 02:30:26.796245 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.796222 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c514c46fd577179a57c3681a35c5ef5aab6d148e58fae47ab874381283144a61" Oct 30 02:30:26.796411 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.796389 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/512be8ad-51e7-4919-a456-9fcfafab46bd-kube-api-access-k44z7\" (UniqueName: \"kubernetes.io/projected/512be8ad-51e7-4919-a456-9fcfafab46bd-kube-api-access-k44z7\") pod \"512be8ad-51e7-4919-a456-9fcfafab46bd\" (UID: \"512be8ad-51e7-4919-a456-9fcfafab46bd\") " Oct 30 02:30:26.796466 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.796449 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/512be8ad-51e7-4919-a456-9fcfafab46bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512be8ad-51e7-4919-a456-9fcfafab46bd-maas-api-service-ca\") pod \"512be8ad-51e7-4919-a456-9fcfafab46bd\" (UID: \"512be8ad-51e7-4919-a456-9fcfafab46bd\") " Oct 30 02:30:26.797381 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.797359 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbr5n\" (UniqueName: \"kubernetes.io/projected/5bce6d68-142e-46ef-b408-89e44ca0fc10-kube-api-access-gbr5n\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.797434 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.797389 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh5lf\" (UniqueName: \"kubernetes.io/projected/3f69dd9b-8eda-4368-b693-33c5b9721710-kube-api-access-hh5lf\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.797434 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.797404 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4lln\" (UniqueName: \"kubernetes.io/projected/860ba768-1529-403e-ba81-13395d6cee83-kube-api-access-n4lln\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.797671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.797646 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512be8ad-51e7-4919-a456-9fcfafab46bd-maas-api-service-ca" pod "512be8ad-51e7-4919-a456-9fcfafab46bd" (UID: "512be8ad-51e7-4919-a456-9fcfafab46bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:26.799943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.799915 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512be8ad-51e7-4919-a456-9fcfafab46bd-kube-api-access-k44z7" pod "512be8ad-51e7-4919-a456-9fcfafab46bd" (UID: "512be8ad-51e7-4919-a456-9fcfafab46bd"). InnerVolumeSpecName "kube-api-access-k44z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:26.859482 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.859445 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" event={"ID":"115cd7ff-80aa-4cd9-a247-9154fe7d117f","Type":"ContainerStarted","Data":"cbadf242590e9e1f74a8d5eb52d2b42d8cc6d497ecfda68be553cd932826dfa8"} Oct 30 02:30:26.860002 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.859980 5998 scope.go:122] "RemoveContainer" containerID="cbadf242590e9e1f74a8d5eb52d2b42d8cc6d497ecfda68be553cd932826dfa8" Oct 30 02:30:26.860938 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:26.860562 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff_odh-ai-gateway-infra(115cd7ff-80aa-4cd9-a247-9154fe7d117f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" podUID="115cd7ff-80aa-4cd9-a247-9154fe7d117f" Oct 30 02:30:26.862683 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.862660 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" event={"ID":"a105635a-7115-4c02-aa79-60516391579b","Type":"ContainerDied","Data":"66d4cb23310319254484218b1151cad3fcc4027edd2df45bcda505fa8c67713d"} Oct 30 02:30:26.862879 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.862860 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt" Oct 30 02:30:26.865784 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.865760 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" event={"ID":"8e81c096-2704-4d9e-80b8-9b1f05e7151f","Type":"ContainerDied","Data":"1e46e96864d9a3f64d99ff69936381ce4eba2dbe0a36b56a092a6e7e423b63b0"} Oct 30 02:30:26.865986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.865968 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z" Oct 30 02:30:26.868026 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.868002 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" Oct 30 02:30:26.868776 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.868149 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh" event={"ID":"3f69dd9b-8eda-4368-b693-33c5b9721710","Type":"ContainerDied","Data":"2105dbefb0bf07c84502095d1c58c9435244f3e82dbfeb979e7230a2a9403970"} Oct 30 02:30:26.868957 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.868906 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn"] Oct 30 02:30:26.879861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.879831 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-584ec5-29888790-548nn"] Oct 30 02:30:26.885729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.885701 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" Oct 30 02:30:26.886564 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.885765 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g" event={"ID":"860ba768-1529-403e-ba81-13395d6cee83","Type":"ContainerDied","Data":"8f9c04e22c707972485b92534f46d7bb0d4c00882459986ee66e3aa855bd7042"} Oct 30 02:30:26.890428 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.890401 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" event={"ID":"1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee","Type":"ContainerDied","Data":"b0fe849dacf17e2b7ce2b4ea60f045619dbbcbbf5ddea7f0d204333d60c15532"} Oct 30 02:30:26.891623 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.891019 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw" Oct 30 02:30:26.895180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.895151 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" event={"ID":"c9b0ea41-0e61-456d-bbce-6ee7145552ea","Type":"ContainerStarted","Data":"a54896aa69689f296b36889a859ff3d4c85a30dca831deb31a36cc2419bcf2e2"} Oct 30 02:30:26.895566 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.895549 5998 scope.go:122] "RemoveContainer" containerID="a54896aa69689f296b36889a859ff3d4c85a30dca831deb31a36cc2419bcf2e2" Oct 30 02:30:26.895827 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:26.895807 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp_odh-ai-gateway-infra(c9b0ea41-0e61-456d-bbce-6ee7145552ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" podUID="c9b0ea41-0e61-456d-bbce-6ee7145552ea" Oct 30 02:30:26.895881 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.895839 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf"] Oct 30 02:30:26.899433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.899410 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k44z7\" (UniqueName: \"kubernetes.io/projected/512be8ad-51e7-4919-a456-9fcfafab46bd-kube-api-access-k44z7\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.899551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.899511 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/512be8ad-51e7-4919-a456-9fcfafab46bd-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:26.968816 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.968781 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" Oct 30 02:30:26.987415 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:26.987378 5998 scope.go:122] "RemoveContainer" containerID="fb366deaa1e6af17dc5d8d44b5a2453a45bd6924de38dc3a5519c08c7e33f769" Oct 30 02:30:27.020115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.004665 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01d4f72c-dc27-4988-b301-873c276855bd-kube-api-access-z2hls\" (UniqueName: \"kubernetes.io/projected/01d4f72c-dc27-4988-b301-873c276855bd-kube-api-access-z2hls\") pod \"01d4f72c-dc27-4988-b301-873c276855bd\" (UID: \"01d4f72c-dc27-4988-b301-873c276855bd\") " Oct 30 02:30:27.020115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.004979 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01d4f72c-dc27-4988-b301-873c276855bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d4f72c-dc27-4988-b301-873c276855bd-maas-api-service-ca\") pod \"01d4f72c-dc27-4988-b301-873c276855bd\" (UID: \"01d4f72c-dc27-4988-b301-873c276855bd\") " Oct 30 02:30:27.020115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.014277 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d4f72c-dc27-4988-b301-873c276855bd-kube-api-access-z2hls" pod "01d4f72c-dc27-4988-b301-873c276855bd" (UID: "01d4f72c-dc27-4988-b301-873c276855bd"). InnerVolumeSpecName "kube-api-access-z2hls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:27.020115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.014665 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d4f72c-dc27-4988-b301-873c276855bd-maas-api-service-ca" pod "01d4f72c-dc27-4988-b301-873c276855bd" (UID: "01d4f72c-dc27-4988-b301-873c276855bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:27.060447 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.060418 5998 scope.go:122] "RemoveContainer" containerID="e21556afbf5d4bc82cf3b0cc5b22a45ff7ef731dd1f7dc496a62fef9c849bd75" Oct 30 02:30:27.064677 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.064638 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q"] Oct 30 02:30:27.070519 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.070490 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c81c1028-xnz6q"] Oct 30 02:30:27.082290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.082064 5998 scope.go:122] "RemoveContainer" containerID="57ff8ac410b8862c3777ce22287520b12cc7f9427839a122162ddab986736a27" Oct 30 02:30:27.144890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.144853 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01d4f72c-dc27-4988-b301-873c276855bd-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:27.145048 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.144891 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2hls\" (UniqueName: \"kubernetes.io/projected/01d4f72c-dc27-4988-b301-873c276855bd-kube-api-access-z2hls\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:27.172668 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.172487 5998 scope.go:122] "RemoveContainer" containerID="beeb1625a64a57cc79961b5b7fa1df2ceae6b5c926f85462418aa49c50a11b81" Oct 30 02:30:27.173620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.173500 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:27.190595 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.185566 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd"] Oct 30 02:30:27.206146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.202485 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d97e4c-29888790-qbxfd"] Oct 30 02:30:27.245953 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.244366 5998 patch_prober.go:29] interesting pod/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 30 02:30:27.245953 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.244438 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" podUID="1d926a5c-800a-4b12-b134-be5c4ff7a619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 30 02:30:27.268108 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.268058 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc"] Oct 30 02:30:27.272615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.272586 5998 scope.go:122] "RemoveContainer" containerID="5c5496d2d5c597896f3eb96b68ab21b73d3d452df02c45c51f049f18c365e299" Oct 30 02:30:27.281622 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.281585 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888790-s2hpc"] Oct 30 02:30:27.289128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.289083 5998 scope.go:122] "RemoveContainer" containerID="8d80588fe34682f48f107a63509767d87845c5cb0b124e117fe8a098be0d579e" Oct 30 02:30:27.298512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.298478 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:27.371287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.371211 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" Oct 30 02:30:27.375320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.375290 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj"] Oct 30 02:30:27.390056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.389717 5998 scope.go:122] "RemoveContainer" containerID="23fd52c0c170e63aef083e4fb54635fb4ffbdc2360a9c1fbd51931c3b3b5abfa" Oct 30 02:30:27.462499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.462470 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0771e950-74b9-45cb-a7be-ccb30c8c8c65-kube-api-access-dx4c2\" (UniqueName: \"kubernetes.io/projected/0771e950-74b9-45cb-a7be-ccb30c8c8c65-kube-api-access-dx4c2\") pod \"0771e950-74b9-45cb-a7be-ccb30c8c8c65\" (UID: \"0771e950-74b9-45cb-a7be-ccb30c8c8c65\") " Oct 30 02:30:27.466954 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.466922 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0771e950-74b9-45cb-a7be-ccb30c8c8c65-kube-api-access-dx4c2" pod "0771e950-74b9-45cb-a7be-ccb30c8c8c65" (UID: "0771e950-74b9-45cb-a7be-ccb30c8c8c65"). InnerVolumeSpecName "kube-api-access-dx4c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:27.471446 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.471422 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0771e950-74b9-45cb-a7be-ccb30c8c8c65-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0771e950-74b9-45cb-a7be-ccb30c8c8c65-maas-api-service-ca\") pod \"0771e950-74b9-45cb-a7be-ccb30c8c8c65\" (UID: \"0771e950-74b9-45cb-a7be-ccb30c8c8c65\") " Oct 30 02:30:27.472655 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.472633 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx4c2\" (UniqueName: \"kubernetes.io/projected/0771e950-74b9-45cb-a7be-ccb30c8c8c65-kube-api-access-dx4c2\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:27.472785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.471839 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0771e950-74b9-45cb-a7be-ccb30c8c8c65-maas-api-service-ca" pod "0771e950-74b9-45cb-a7be-ccb30c8c8c65" (UID: "0771e950-74b9-45cb-a7be-ccb30c8c8c65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:27.475890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.475863 5998 scope.go:122] "RemoveContainer" containerID="a46d34a6e347b327c9bd96ffb9082e2dd297f3d238ab0439465bfaeb7a10a0a8" Oct 30 02:30:27.480188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.480159 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k"] Oct 30 02:30:27.487729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.487214 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888790-mgt8k"] Oct 30 02:30:27.492118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.492038 5998 scope.go:122] "RemoveContainer" containerID="9c6aecc1135c9ccc3da29f3c2cea710859113a733187cba4cdc6286468b3a836" Oct 30 02:30:27.548222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.548193 5998 scope.go:122] "RemoveContainer" containerID="eccb5b86e47fe184c5d48c9ac012c1a64c63b0ea808dddb73fff7ea558698cac" Oct 30 02:30:27.560353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.560332 5998 scope.go:122] "RemoveContainer" containerID="d73756e131cde7b384c390a639c3624c9e5055480ff0b03323e4ebe27f4039bb" Oct 30 02:30:27.575672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.575595 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0771e950-74b9-45cb-a7be-ccb30c8c8c65-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:27.651697 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.651609 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt"] Oct 30 02:30:27.660181 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.660083 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93ec46-29888790-f2ztt"] Oct 30 02:30:27.683831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.683802 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z"] Oct 30 02:30:27.684916 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.684892 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" Oct 30 02:30:27.725009 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.724961 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3d5da6-29888790-rfb5z"] Oct 30 02:30:27.777753 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.777721 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96bc88e1-7b35-4b54-b12f-69227ae41a88-kube-api-access-v4cvr\" (UniqueName: \"kubernetes.io/projected/96bc88e1-7b35-4b54-b12f-69227ae41a88-kube-api-access-v4cvr\") pod \"96bc88e1-7b35-4b54-b12f-69227ae41a88\" (UID: \"96bc88e1-7b35-4b54-b12f-69227ae41a88\") " Oct 30 02:30:27.777944 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.777915 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96bc88e1-7b35-4b54-b12f-69227ae41a88-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bc88e1-7b35-4b54-b12f-69227ae41a88-maas-api-service-ca\") pod \"96bc88e1-7b35-4b54-b12f-69227ae41a88\" (UID: \"96bc88e1-7b35-4b54-b12f-69227ae41a88\") " Oct 30 02:30:27.778303 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.778277 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96bc88e1-7b35-4b54-b12f-69227ae41a88-maas-api-service-ca" pod "96bc88e1-7b35-4b54-b12f-69227ae41a88" (UID: "96bc88e1-7b35-4b54-b12f-69227ae41a88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:27.778637 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.778616 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96bc88e1-7b35-4b54-b12f-69227ae41a88-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:27.779845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.779825 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96bc88e1-7b35-4b54-b12f-69227ae41a88-kube-api-access-v4cvr" pod "96bc88e1-7b35-4b54-b12f-69227ae41a88" (UID: "96bc88e1-7b35-4b54-b12f-69227ae41a88"). InnerVolumeSpecName "kube-api-access-v4cvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:27.852633 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.852586 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr"] Oct 30 02:30:27.872454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.872418 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888790-zg5sr"] Oct 30 02:30:27.880054 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.880015 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4cvr\" (UniqueName: \"kubernetes.io/projected/96bc88e1-7b35-4b54-b12f-69227ae41a88-kube-api-access-v4cvr\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:27.940883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.940843 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq" Oct 30 02:30:27.943588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.943559 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" Oct 30 02:30:27.943834 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.943562 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj" event={"ID":"96bc88e1-7b35-4b54-b12f-69227ae41a88","Type":"ContainerDied","Data":"4e96c5d1588076e5dc8703de7d0a0c90023fc4a602129ea39a8e21abac9e146a"} Oct 30 02:30:27.943971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.943954 5998 scope.go:122] "RemoveContainer" containerID="0bae572f93f41aff92d8c004345ded2c236edc1b4fe05fa96089f74fd2c92cfd" Oct 30 02:30:27.959263 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.953405 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" event={"ID":"0771e950-74b9-45cb-a7be-ccb30c8c8c65","Type":"ContainerDied","Data":"3bf9b239f0b8313a68f73a1ef111e1da1b76a1896fddb9abd828fdc4d6ea2d79"} Oct 30 02:30:27.959263 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.953518 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf" Oct 30 02:30:27.959473 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.959288 5998 generic.go:356] "Generic (PLEG): container finished" podID="9e4909d0-f3ae-423a-a478-4ee5ce9734a1" containerID="7e96b738e05652e8598fca8c9e6f4393fd7c9551cb8e37976c0ea08a6843a528" exitCode=60 Oct 30 02:30:27.959473 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.959396 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" event={"ID":"9e4909d0-f3ae-423a-a478-4ee5ce9734a1","Type":"ContainerDied","Data":"7e96b738e05652e8598fca8c9e6f4393fd7c9551cb8e37976c0ea08a6843a528"} Oct 30 02:30:27.960163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.959929 5998 scope.go:122] "RemoveContainer" containerID="7e96b738e05652e8598fca8c9e6f4393fd7c9551cb8e37976c0ea08a6843a528" Oct 30 02:30:27.960517 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:27.960398 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc_odh-ai-gateway-infra(9e4909d0-f3ae-423a-a478-4ee5ce9734a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" podUID="9e4909d0-f3ae-423a-a478-4ee5ce9734a1" Oct 30 02:30:27.960517 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.960456 5998 scope.go:122] "RemoveContainer" containerID="e7863ad2ff142bff058f26383400d5c2e71aa4c832457ba9f0e0bbb88a612d4b" Oct 30 02:30:27.964132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.964079 5998 generic.go:356] "Generic (PLEG): container finished" podID="daaa2b88-a49a-4a88-8325-33307e969074" containerID="0f64758a273c7816f2df6c0f9c08c7a5aa2ca0d4e0b784affbab53611ad05ff9" exitCode=60 Oct 30 02:30:27.964273 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.964253 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" event={"ID":"daaa2b88-a49a-4a88-8325-33307e969074","Type":"ContainerDied","Data":"0f64758a273c7816f2df6c0f9c08c7a5aa2ca0d4e0b784affbab53611ad05ff9"} Oct 30 02:30:27.964861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.964845 5998 scope.go:122] "RemoveContainer" containerID="0f64758a273c7816f2df6c0f9c08c7a5aa2ca0d4e0b784affbab53611ad05ff9" Oct 30 02:30:27.965415 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:27.965396 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2_odh-ai-gateway-infra(daaa2b88-a49a-4a88-8325-33307e969074)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" podUID="daaa2b88-a49a-4a88-8325-33307e969074" Oct 30 02:30:27.966388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.966351 5998 generic.go:356] "Generic (PLEG): container finished" podID="115cd7ff-80aa-4cd9-a247-9154fe7d117f" containerID="cbadf242590e9e1f74a8d5eb52d2b42d8cc6d497ecfda68be553cd932826dfa8" exitCode=60 Oct 30 02:30:27.966483 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.966463 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" event={"ID":"115cd7ff-80aa-4cd9-a247-9154fe7d117f","Type":"ContainerDied","Data":"cbadf242590e9e1f74a8d5eb52d2b42d8cc6d497ecfda68be553cd932826dfa8"} Oct 30 02:30:27.966872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.966853 5998 scope.go:122] "RemoveContainer" containerID="cbadf242590e9e1f74a8d5eb52d2b42d8cc6d497ecfda68be553cd932826dfa8" Oct 30 02:30:27.967616 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:27.967268 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff_odh-ai-gateway-infra(115cd7ff-80aa-4cd9-a247-9154fe7d117f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" podUID="115cd7ff-80aa-4cd9-a247-9154fe7d117f" Oct 30 02:30:27.974099 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.974064 5998 generic.go:356] "Generic (PLEG): container finished" podID="c9b0ea41-0e61-456d-bbce-6ee7145552ea" containerID="a54896aa69689f296b36889a859ff3d4c85a30dca831deb31a36cc2419bcf2e2" exitCode=60 Oct 30 02:30:27.974184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.974116 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" event={"ID":"c9b0ea41-0e61-456d-bbce-6ee7145552ea","Type":"ContainerDied","Data":"a54896aa69689f296b36889a859ff3d4c85a30dca831deb31a36cc2419bcf2e2"} Oct 30 02:30:27.974227 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.974197 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw" Oct 30 02:30:27.974557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.974540 5998 scope.go:122] "RemoveContainer" containerID="a54896aa69689f296b36889a859ff3d4c85a30dca831deb31a36cc2419bcf2e2" Oct 30 02:30:27.974860 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:27.974838 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp_odh-ai-gateway-infra(c9b0ea41-0e61-456d-bbce-6ee7145552ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" podUID="c9b0ea41-0e61-456d-bbce-6ee7145552ea" Oct 30 02:30:27.979552 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.979533 5998 scope.go:122] "RemoveContainer" containerID="bfe24a95fde280b39bfc982bd812c048dbe52ef1d7489db9b9f45ea694d55eb1" Oct 30 02:30:27.990070 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.990046 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g"] Oct 30 02:30:27.994576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:27.994552 5998 scope.go:122] "RemoveContainer" containerID="4808839e8ecf47d51616536ac0709d0b7dff0daac511e5a59c6ce6f9dbe81f12" Oct 30 02:30:28.033416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.033387 5998 scope.go:122] "RemoveContainer" containerID="55cba0eb40314867cf6b2d66f160469b841bb0c32284f6987eafd0778727b966" Oct 30 02:30:28.038849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.038819 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1fddb7-29888790-h9q8g"] Oct 30 02:30:28.048218 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.048193 5998 scope.go:122] "RemoveContainer" containerID="e3cd7f2ab448a9fb591dcb3c307bc71ea1ee8373a79b61aa5a92080b6704e74e" Oct 30 02:30:28.088392 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.088366 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw"] Oct 30 02:30:28.128271 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.128232 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-0ce950-29888790-mbzxw"] Oct 30 02:30:28.183863 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.183829 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh"] Oct 30 02:30:28.190755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.190699 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d207e8-29888790-xndqh"] Oct 30 02:30:28.246274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.246182 5998 patch_prober.go:29] interesting pod/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 30 02:30:28.246274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.246246 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" podUID="1d926a5c-800a-4b12-b134-be5c4ff7a619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 30 02:30:28.258744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.258705 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6"] Oct 30 02:30:28.265735 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.265703 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888790-ltjr6"] Oct 30 02:30:28.289139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.289100 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:28.474163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.474126 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf"] Oct 30 02:30:28.483941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.483868 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-b93d51-29888790-47fmf"] Oct 30 02:30:28.551316 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.551195 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0771e950-74b9-45cb-a7be-ccb30c8c8c65" path="/var/lib/kubelet/pods/0771e950-74b9-45cb-a7be-ccb30c8c8c65/volumes" Oct 30 02:30:28.551793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.551774 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee" path="/var/lib/kubelet/pods/1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee/volumes" Oct 30 02:30:28.552268 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.552249 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb" path="/var/lib/kubelet/pods/3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb/volumes" Oct 30 02:30:28.553040 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.553019 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f69dd9b-8eda-4368-b693-33c5b9721710" path="/var/lib/kubelet/pods/3f69dd9b-8eda-4368-b693-33c5b9721710/volumes" Oct 30 02:30:28.553490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.553473 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="512d72f9-4883-4d39-9df8-9282f56da2ac" path="/var/lib/kubelet/pods/512d72f9-4883-4d39-9df8-9282f56da2ac/volumes" Oct 30 02:30:28.554133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.554071 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bce6d68-142e-46ef-b408-89e44ca0fc10" path="/var/lib/kubelet/pods/5bce6d68-142e-46ef-b408-89e44ca0fc10/volumes" Oct 30 02:30:28.554886 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.554866 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df3039d-5102-404f-b58f-eae1e79feb36" path="/var/lib/kubelet/pods/6df3039d-5102-404f-b58f-eae1e79feb36/volumes" Oct 30 02:30:28.555363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.555346 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="860ba768-1529-403e-ba81-13395d6cee83" path="/var/lib/kubelet/pods/860ba768-1529-403e-ba81-13395d6cee83/volumes" Oct 30 02:30:28.555833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.555813 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e81c096-2704-4d9e-80b8-9b1f05e7151f" path="/var/lib/kubelet/pods/8e81c096-2704-4d9e-80b8-9b1f05e7151f/volumes" Oct 30 02:30:28.557249 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.557229 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a105635a-7115-4c02-aa79-60516391579b" path="/var/lib/kubelet/pods/a105635a-7115-4c02-aa79-60516391579b/volumes" Oct 30 02:30:28.557662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.557646 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1f42aec-2169-46f1-a837-039b920ac828" path="/var/lib/kubelet/pods/a1f42aec-2169-46f1-a837-039b920ac828/volumes" Oct 30 02:30:28.558407 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.558386 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9c846c2-743f-49ca-88e8-49a4936bfe23" path="/var/lib/kubelet/pods/a9c846c2-743f-49ca-88e8-49a4936bfe23/volumes" Oct 30 02:30:28.558821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.558801 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be309ab3-73d7-42c0-92a8-0fc686504795" path="/var/lib/kubelet/pods/be309ab3-73d7-42c0-92a8-0fc686504795/volumes" Oct 30 02:30:28.559274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.559252 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5882ea9-fe2e-4587-abb7-bb14939f53f6" path="/var/lib/kubelet/pods/e5882ea9-fe2e-4587-abb7-bb14939f53f6/volumes" Oct 30 02:30:28.559668 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.559652 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs"] Oct 30 02:30:28.559736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.559709 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw"] Oct 30 02:30:28.559736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.559724 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-db01c4-29888790-vd4xw"] Oct 30 02:30:28.561529 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.561505 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp"] Oct 30 02:30:28.668509 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.668442 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq"] Oct 30 02:30:28.674975 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.674939 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2943ff-29888790-ncpdq"] Oct 30 02:30:28.749374 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.749340 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj"] Oct 30 02:30:28.756735 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.756710 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888790-xcsmj"] Oct 30 02:30:28.763958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.763925 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2"] Oct 30 02:30:28.831163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.830534 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" Oct 30 02:30:28.882562 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.882525 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f3ead60-88ca-42af-9e25-7822589058e4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3ead60-88ca-42af-9e25-7822589058e4-maas-api-service-ca\") pod \"2f3ead60-88ca-42af-9e25-7822589058e4\" (UID: \"2f3ead60-88ca-42af-9e25-7822589058e4\") " Oct 30 02:30:28.882938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.882909 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3ead60-88ca-42af-9e25-7822589058e4-maas-api-service-ca" pod "2f3ead60-88ca-42af-9e25-7822589058e4" (UID: "2f3ead60-88ca-42af-9e25-7822589058e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:28.883074 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.883053 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f3ead60-88ca-42af-9e25-7822589058e4-kube-api-access-gjdxg\" (UniqueName: \"kubernetes.io/projected/2f3ead60-88ca-42af-9e25-7822589058e4-kube-api-access-gjdxg\") pod \"2f3ead60-88ca-42af-9e25-7822589058e4\" (UID: \"2f3ead60-88ca-42af-9e25-7822589058e4\") " Oct 30 02:30:28.883901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.883878 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3ead60-88ca-42af-9e25-7822589058e4-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:28.885905 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.885878 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3ead60-88ca-42af-9e25-7822589058e4-kube-api-access-gjdxg" pod "2f3ead60-88ca-42af-9e25-7822589058e4" (UID: "2f3ead60-88ca-42af-9e25-7822589058e4"). InnerVolumeSpecName "kube-api-access-gjdxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:28.932611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.932562 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc"] Oct 30 02:30:28.985765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.985720 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjdxg\" (UniqueName: \"kubernetes.io/projected/2f3ead60-88ca-42af-9e25-7822589058e4-kube-api-access-gjdxg\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:28.987334 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.987311 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" Oct 30 02:30:28.987334 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.987319 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs" event={"ID":"2f3ead60-88ca-42af-9e25-7822589058e4","Type":"ContainerDied","Data":"226f6f99f1eabfe31ac2b59e1494cb518c0b315f67d29890719a3a3963ab6b0c"} Oct 30 02:30:28.987481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:28.987385 5998 scope.go:122] "RemoveContainer" containerID="6c7f23c8f09d2384f01c1c24c8e3adaeed5293f578e083da2ede26f06f312787" Oct 30 02:30:29.062798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.062764 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs"] Oct 30 02:30:29.077176 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.076337 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888790-dp8qs"] Oct 30 02:30:29.161950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.161910 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff"] Oct 30 02:30:29.244783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.244741 5998 patch_prober.go:29] interesting pod/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" start-of-body= Oct 30 02:30:29.245044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.245017 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" podUID="1d926a5c-800a-4b12-b134-be5c4ff7a619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.27:15021/healthz/ready\": dial tcp 10.133.1.27:15021: connect: connection refused" Oct 30 02:30:29.330833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.330798 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" Oct 30 02:30:29.388719 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.388671 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:29.416083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.416044 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daaa2b88-a49a-4a88-8325-33307e969074-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daaa2b88-a49a-4a88-8325-33307e969074-maas-api-service-ca\") pod \"daaa2b88-a49a-4a88-8325-33307e969074\" (UID: \"daaa2b88-a49a-4a88-8325-33307e969074\") " Oct 30 02:30:29.417102 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.417062 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daaa2b88-a49a-4a88-8325-33307e969074-kube-api-access-cmtcf\" (UniqueName: \"kubernetes.io/projected/daaa2b88-a49a-4a88-8325-33307e969074-kube-api-access-cmtcf\") pod \"daaa2b88-a49a-4a88-8325-33307e969074\" (UID: \"daaa2b88-a49a-4a88-8325-33307e969074\") " Oct 30 02:30:29.418296 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.416874 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daaa2b88-a49a-4a88-8325-33307e969074-maas-api-service-ca" pod "daaa2b88-a49a-4a88-8325-33307e969074" (UID: "daaa2b88-a49a-4a88-8325-33307e969074"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:29.418424 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.416503 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" Oct 30 02:30:29.459968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.459912 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daaa2b88-a49a-4a88-8325-33307e969074-kube-api-access-cmtcf" pod "daaa2b88-a49a-4a88-8325-33307e969074" (UID: "daaa2b88-a49a-4a88-8325-33307e969074"). InnerVolumeSpecName "kube-api-access-cmtcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:29.519807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.519773 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e4909d0-f3ae-423a-a478-4ee5ce9734a1-kube-api-access-clr67\" (UniqueName: \"kubernetes.io/projected/9e4909d0-f3ae-423a-a478-4ee5ce9734a1-kube-api-access-clr67\") pod \"9e4909d0-f3ae-423a-a478-4ee5ce9734a1\" (UID: \"9e4909d0-f3ae-423a-a478-4ee5ce9734a1\") " Oct 30 02:30:29.519998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.519860 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e4909d0-f3ae-423a-a478-4ee5ce9734a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e4909d0-f3ae-423a-a478-4ee5ce9734a1-maas-api-service-ca\") pod \"9e4909d0-f3ae-423a-a478-4ee5ce9734a1\" (UID: \"9e4909d0-f3ae-423a-a478-4ee5ce9734a1\") " Oct 30 02:30:29.521751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.520681 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daaa2b88-a49a-4a88-8325-33307e969074-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:29.521751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.520711 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmtcf\" (UniqueName: \"kubernetes.io/projected/daaa2b88-a49a-4a88-8325-33307e969074-kube-api-access-cmtcf\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:29.521751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.521118 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e4909d0-f3ae-423a-a478-4ee5ce9734a1-maas-api-service-ca" pod "9e4909d0-f3ae-423a-a478-4ee5ce9734a1" (UID: "9e4909d0-f3ae-423a-a478-4ee5ce9734a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:29.526781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.526379 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" Oct 30 02:30:29.526781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.526661 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4909d0-f3ae-423a-a478-4ee5ce9734a1-kube-api-access-clr67" pod "9e4909d0-f3ae-423a-a478-4ee5ce9734a1" (UID: "9e4909d0-f3ae-423a-a478-4ee5ce9734a1"). InnerVolumeSpecName "kube-api-access-clr67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:29.538142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.536723 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:29.576480 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.576113 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" Oct 30 02:30:29.623266 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.623156 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b0ea41-0e61-456d-bbce-6ee7145552ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b0ea41-0e61-456d-bbce-6ee7145552ea-maas-api-service-ca\") pod \"c9b0ea41-0e61-456d-bbce-6ee7145552ea\" (UID: \"c9b0ea41-0e61-456d-bbce-6ee7145552ea\") " Oct 30 02:30:29.624380 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.624349 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b0ea41-0e61-456d-bbce-6ee7145552ea-kube-api-access-jkcpq\" (UniqueName: \"kubernetes.io/projected/c9b0ea41-0e61-456d-bbce-6ee7145552ea-kube-api-access-jkcpq\") pod \"c9b0ea41-0e61-456d-bbce-6ee7145552ea\" (UID: \"c9b0ea41-0e61-456d-bbce-6ee7145552ea\") " Oct 30 02:30:29.624847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.624826 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115cd7ff-80aa-4cd9-a247-9154fe7d117f-kube-api-access-vkvfw\" (UniqueName: \"kubernetes.io/projected/115cd7ff-80aa-4cd9-a247-9154fe7d117f-kube-api-access-vkvfw\") pod \"115cd7ff-80aa-4cd9-a247-9154fe7d117f\" (UID: \"115cd7ff-80aa-4cd9-a247-9154fe7d117f\") " Oct 30 02:30:29.625156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.625130 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b0ea41-0e61-456d-bbce-6ee7145552ea-maas-api-service-ca" pod "c9b0ea41-0e61-456d-bbce-6ee7145552ea" (UID: "c9b0ea41-0e61-456d-bbce-6ee7145552ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:29.625301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.625284 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/115cd7ff-80aa-4cd9-a247-9154fe7d117f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/115cd7ff-80aa-4cd9-a247-9154fe7d117f-maas-api-service-ca\") pod \"115cd7ff-80aa-4cd9-a247-9154fe7d117f\" (UID: \"115cd7ff-80aa-4cd9-a247-9154fe7d117f\") " Oct 30 02:30:29.627588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.627566 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b0ea41-0e61-456d-bbce-6ee7145552ea-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:29.627729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.627716 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clr67\" (UniqueName: \"kubernetes.io/projected/9e4909d0-f3ae-423a-a478-4ee5ce9734a1-kube-api-access-clr67\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:29.627842 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.627829 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e4909d0-f3ae-423a-a478-4ee5ce9734a1-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:29.635606 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.635535 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115cd7ff-80aa-4cd9-a247-9154fe7d117f-maas-api-service-ca" pod "115cd7ff-80aa-4cd9-a247-9154fe7d117f" (UID: "115cd7ff-80aa-4cd9-a247-9154fe7d117f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:29.637932 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.637885 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b0ea41-0e61-456d-bbce-6ee7145552ea-kube-api-access-jkcpq" pod "c9b0ea41-0e61-456d-bbce-6ee7145552ea" (UID: "c9b0ea41-0e61-456d-bbce-6ee7145552ea"). InnerVolumeSpecName "kube-api-access-jkcpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:29.641044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.640997 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115cd7ff-80aa-4cd9-a247-9154fe7d117f-kube-api-access-vkvfw" pod "115cd7ff-80aa-4cd9-a247-9154fe7d117f" (UID: "115cd7ff-80aa-4cd9-a247-9154fe7d117f"). InnerVolumeSpecName "kube-api-access-vkvfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:29.780762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.780717 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkvfw\" (UniqueName: \"kubernetes.io/projected/115cd7ff-80aa-4cd9-a247-9154fe7d117f-kube-api-access-vkvfw\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:29.789115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.781061 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/115cd7ff-80aa-4cd9-a247-9154fe7d117f-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:29.802114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:29.789355 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkcpq\" (UniqueName: \"kubernetes.io/projected/c9b0ea41-0e61-456d-bbce-6ee7145552ea-kube-api-access-jkcpq\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:30.066819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.066779 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" event={"ID":"9e4909d0-f3ae-423a-a478-4ee5ce9734a1","Type":"ContainerDied","Data":"6dfed4a59b4d7ed707d77f54a717104f003339736e547b12758b60d6801f72ad"} Oct 30 02:30:30.076221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.076180 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" event={"ID":"daaa2b88-a49a-4a88-8325-33307e969074","Type":"ContainerDied","Data":"6ae2cb2102a68c0bdd80c4b22a45b2edc3ed8ad5e88413a988280ed61a139d0f"} Oct 30 02:30:30.077245 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.077216 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" event={"ID":"115cd7ff-80aa-4cd9-a247-9154fe7d117f","Type":"ContainerDied","Data":"8e281291caa5ea8f17eea9b50df8e516bf91b7e76d023c3562512865aee1179f"} Oct 30 02:30:30.077465 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.077448 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" event={"ID":"c9b0ea41-0e61-456d-bbce-6ee7145552ea","Type":"ContainerDied","Data":"d12e5e487d443806e4fd2e70fb6f93faf5a798386673f7171cbe6e5094afb2ad"} Oct 30 02:30:30.077622 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.067081 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc" Oct 30 02:30:30.080145 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.076462 5998 scope.go:122] "RemoveContainer" containerID="7e96b738e05652e8598fca8c9e6f4393fd7c9551cb8e37976c0ea08a6843a528" Oct 30 02:30:30.080368 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.077042 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp" Oct 30 02:30:30.081064 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.069025 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2" Oct 30 02:30:30.084139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.071189 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff" Oct 30 02:30:30.102909 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.102876 5998 scope.go:122] "RemoveContainer" containerID="0f64758a273c7816f2df6c0f9c08c7a5aa2ca0d4e0b784affbab53611ad05ff9" Oct 30 02:30:30.124282 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.124256 5998 scope.go:122] "RemoveContainer" containerID="cbadf242590e9e1f74a8d5eb52d2b42d8cc6d497ecfda68be553cd932826dfa8" Oct 30 02:30:30.138636 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.138598 5998 scope.go:122] "RemoveContainer" containerID="a54896aa69689f296b36889a859ff3d4c85a30dca831deb31a36cc2419bcf2e2" Oct 30 02:30:30.144074 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.144028 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2"] Oct 30 02:30:30.155247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.155213 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-9dfb3c-29888790-nlwl2"] Oct 30 02:30:30.190624 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.190587 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff"] Oct 30 02:30:30.201572 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.201534 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-17e747-29888790-47pff"] Oct 30 02:30:30.218877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.218837 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:30.228185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.228149 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp"] Oct 30 02:30:30.233220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.233177 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-000edd-29888790-ktqsp"] Oct 30 02:30:30.318900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.318797 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc"] Oct 30 02:30:30.344500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.344462 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-52d131-29888790-hd2lc"] Oct 30 02:30:30.400460 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.400375 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:30.401162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.401131 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:30.403558 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.403529 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:30:30.544665 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.544629 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01d4f72c-dc27-4988-b301-873c276855bd" path="/var/lib/kubelet/pods/01d4f72c-dc27-4988-b301-873c276855bd/volumes" Oct 30 02:30:30.545179 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.545162 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="115cd7ff-80aa-4cd9-a247-9154fe7d117f" path="/var/lib/kubelet/pods/115cd7ff-80aa-4cd9-a247-9154fe7d117f/volumes" Oct 30 02:30:30.545640 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.545623 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f3ead60-88ca-42af-9e25-7822589058e4" path="/var/lib/kubelet/pods/2f3ead60-88ca-42af-9e25-7822589058e4/volumes" Oct 30 02:30:30.546382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.546363 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="512be8ad-51e7-4919-a456-9fcfafab46bd" path="/var/lib/kubelet/pods/512be8ad-51e7-4919-a456-9fcfafab46bd/volumes" Oct 30 02:30:30.546816 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.546798 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96bc88e1-7b35-4b54-b12f-69227ae41a88" path="/var/lib/kubelet/pods/96bc88e1-7b35-4b54-b12f-69227ae41a88/volumes" Oct 30 02:30:30.547248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.547232 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e4909d0-f3ae-423a-a478-4ee5ce9734a1" path="/var/lib/kubelet/pods/9e4909d0-f3ae-423a-a478-4ee5ce9734a1/volumes" Oct 30 02:30:30.547963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.547944 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9b0ea41-0e61-456d-bbce-6ee7145552ea" path="/var/lib/kubelet/pods/c9b0ea41-0e61-456d-bbce-6ee7145552ea/volumes" Oct 30 02:30:30.548374 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:30.548359 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daaa2b88-a49a-4a88-8325-33307e969074" path="/var/lib/kubelet/pods/daaa2b88-a49a-4a88-8325-33307e969074/volumes" Oct 30 02:30:31.885129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:31.885063 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:33.410541 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:33.410505 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:33.891182 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:33.891076 5998 scope.go:122] "RemoveContainer" containerID="969ed54ed780136ee97e43950a6c0a77f61572f0e4b9522b67f3020a6e2f126e" Oct 30 02:30:35.000847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:35.000797 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:36.624486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:36.624412 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:38.224115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:38.220468 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:39.547475 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:39.547423 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:39.911049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:39.911013 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:41.545614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:41.545574 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:42.733126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:42.733074 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c"] Oct 30 02:30:42.733660 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:42.733331 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" podUID="21de7884-5d25-40b9-9cc4-5f9f87eff813" containerName="discovery" containerID="cri-o://c420fbd1911f4930076a1c63a1bd3a9e1137fa4d4bdc0a515831d97f38732fcb" gracePeriod=30 Oct 30 02:30:43.063133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.063073 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:30:43.096164 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.096126 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:43.133901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.133848 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-csr-dns-cert\") pod \"21de7884-5d25-40b9-9cc4-5f9f87eff813\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " Oct 30 02:30:43.134084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.133999 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21de7884-5d25-40b9-9cc4-5f9f87eff813-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21de7884-5d25-40b9-9cc4-5f9f87eff813-local-certs\") pod \"21de7884-5d25-40b9-9cc4-5f9f87eff813\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " Oct 30 02:30:43.134202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.134171 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21de7884-5d25-40b9-9cc4-5f9f87eff813-kube-api-access-r29g8\" (UniqueName: \"kubernetes.io/projected/21de7884-5d25-40b9-9cc4-5f9f87eff813-kube-api-access-r29g8\") pod \"21de7884-5d25-40b9-9cc4-5f9f87eff813\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " Oct 30 02:30:43.134279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.134221 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-csr-ca-configmap\") pod \"21de7884-5d25-40b9-9cc4-5f9f87eff813\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " Oct 30 02:30:43.134363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.134295 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-token\" (UniqueName: \"kubernetes.io/projected/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-token\") pod \"21de7884-5d25-40b9-9cc4-5f9f87eff813\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " Oct 30 02:30:43.135040 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.135010 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-kubeconfig\") pod \"21de7884-5d25-40b9-9cc4-5f9f87eff813\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " Oct 30 02:30:43.135224 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.135137 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-cacerts\" (UniqueName: \"kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-cacerts\") pod \"21de7884-5d25-40b9-9cc4-5f9f87eff813\" (UID: \"21de7884-5d25-40b9-9cc4-5f9f87eff813\") " Oct 30 02:30:43.135224 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.135132 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-csr-ca-configmap" pod "21de7884-5d25-40b9-9cc4-5f9f87eff813" (UID: "21de7884-5d25-40b9-9cc4-5f9f87eff813"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:30:43.136568 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.136541 5998 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-csr-ca-configmap\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:43.144363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.144313 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21de7884-5d25-40b9-9cc4-5f9f87eff813-local-certs" pod "21de7884-5d25-40b9-9cc4-5f9f87eff813" (UID: "21de7884-5d25-40b9-9cc4-5f9f87eff813"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:30:43.148464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.148414 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-token" pod "21de7884-5d25-40b9-9cc4-5f9f87eff813" (UID: "21de7884-5d25-40b9-9cc4-5f9f87eff813"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:43.148710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.148692 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-kubeconfig" pod "21de7884-5d25-40b9-9cc4-5f9f87eff813" (UID: "21de7884-5d25-40b9-9cc4-5f9f87eff813"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:30:43.148922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.148906 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21de7884-5d25-40b9-9cc4-5f9f87eff813-kube-api-access-r29g8" pod "21de7884-5d25-40b9-9cc4-5f9f87eff813" (UID: "21de7884-5d25-40b9-9cc4-5f9f87eff813"). InnerVolumeSpecName "kube-api-access-r29g8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:30:43.150377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.150218 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-csr-dns-cert" pod "21de7884-5d25-40b9-9cc4-5f9f87eff813" (UID: "21de7884-5d25-40b9-9cc4-5f9f87eff813"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:30:43.151647 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.151612 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-cacerts" pod "21de7884-5d25-40b9-9cc4-5f9f87eff813" (UID: "21de7884-5d25-40b9-9cc4-5f9f87eff813"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:30:43.203879 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.203841 5998 generic.go:356] "Generic (PLEG): container finished" podID="21de7884-5d25-40b9-9cc4-5f9f87eff813" containerID="c420fbd1911f4930076a1c63a1bd3a9e1137fa4d4bdc0a515831d97f38732fcb" exitCode=0 Oct 30 02:30:43.204053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.203911 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" Oct 30 02:30:43.204053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.203918 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" event={"ID":"21de7884-5d25-40b9-9cc4-5f9f87eff813","Type":"ContainerDied","Data":"c420fbd1911f4930076a1c63a1bd3a9e1137fa4d4bdc0a515831d97f38732fcb"} Oct 30 02:30:43.204053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.204010 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c" event={"ID":"21de7884-5d25-40b9-9cc4-5f9f87eff813","Type":"ContainerDied","Data":"1500781dfc5f2096af685fb2fe7d68694ba5d1915e8de625de6b87a650e05ae7"} Oct 30 02:30:43.204053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.204039 5998 scope.go:122] "RemoveContainer" containerID="c420fbd1911f4930076a1c63a1bd3a9e1137fa4d4bdc0a515831d97f38732fcb" Oct 30 02:30:43.253029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.252905 5998 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-kubeconfig\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:43.253029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.252944 5998 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-cacerts\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:43.253029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.252960 5998 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-csr-dns-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:43.253029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.252974 5998 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/21de7884-5d25-40b9-9cc4-5f9f87eff813-local-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:43.253029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.252988 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r29g8\" (UniqueName: \"kubernetes.io/projected/21de7884-5d25-40b9-9cc4-5f9f87eff813-kube-api-access-r29g8\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:43.253029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.253002 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21de7884-5d25-40b9-9cc4-5f9f87eff813-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:30:43.298010 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.297080 5998 scope.go:122] "RemoveContainer" containerID="c420fbd1911f4930076a1c63a1bd3a9e1137fa4d4bdc0a515831d97f38732fcb" Oct 30 02:30:43.298187 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:43.298084 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c420fbd1911f4930076a1c63a1bd3a9e1137fa4d4bdc0a515831d97f38732fcb\": container with ID starting with c420fbd1911f4930076a1c63a1bd3a9e1137fa4d4bdc0a515831d97f38732fcb not found: ID does not exist" containerID="c420fbd1911f4930076a1c63a1bd3a9e1137fa4d4bdc0a515831d97f38732fcb" Oct 30 02:30:43.298251 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.298184 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c420fbd1911f4930076a1c63a1bd3a9e1137fa4d4bdc0a515831d97f38732fcb"} err="failed to get container status \"c420fbd1911f4930076a1c63a1bd3a9e1137fa4d4bdc0a515831d97f38732fcb\": rpc error: code = NotFound desc = could not find container \"c420fbd1911f4930076a1c63a1bd3a9e1137fa4d4bdc0a515831d97f38732fcb\": container with ID starting with c420fbd1911f4930076a1c63a1bd3a9e1137fa4d4bdc0a515831d97f38732fcb not found: ID does not exist" Oct 30 02:30:43.304000 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.303966 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c"] Oct 30 02:30:43.310641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:43.310607 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-sc85c"] Oct 30 02:30:44.541523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:44.541487 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21de7884-5d25-40b9-9cc4-5f9f87eff813" path="/var/lib/kubelet/pods/21de7884-5d25-40b9-9cc4-5f9f87eff813/volumes" Oct 30 02:30:44.723812 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:44.723776 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:46.243610 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:46.243576 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:47.891680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:47.891641 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:49.433231 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:49.433195 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:50.979405 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:50.979366 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:52.816574 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:52.816530 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:54.506899 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:54.506860 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:55.224439 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:30:55.224399 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:30:56.292584 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:56.292544 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:57.995785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:57.995746 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:30:59.568665 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:30:59.568622 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:01.241183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:01.241141 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:02.759057 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:02.759017 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:04.388267 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:04.388232 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:06.028416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:06.028376 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:07.594432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:07.594387 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:09.111309 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:09.111248 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:10.806323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:10.806284 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:12.476568 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:12.476474 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:14.041904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:14.041825 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:15.693998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:15.693965 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:17.233831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:17.233792 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:17.990659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:17.990623 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh"] Oct 30 02:31:17.990939 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:17.990846 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" podUID="7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" containerName="istio-proxy" containerID="cri-o://84c0ff3b7967a3cfda918ac128b2b0c927359ef6f7a01a5a49fc77efdb625b73" gracePeriod=30 Oct 30 02:31:18.001753 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:18.001721 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v"] Oct 30 02:31:18.002326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:18.002290 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" podUID="caaec13d-e62d-4450-864c-d9e3fb7ca8d4" containerName="istio-proxy" containerID="cri-o://dd5450b72b56f3408b6dd3dd3799b0306b2253d06e54c4d3421bd5ecd1bcfd4e" gracePeriod=30 Oct 30 02:31:18.005662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:18.005622 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7"] Oct 30 02:31:18.005856 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:18.005832 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" podUID="9782ca9d-bd87-415c-bad8-e080814048e7" containerName="istio-proxy" containerID="cri-o://d0d222cc228a1afd865c05be868c3014eb31cfafa83a82c0e44d46117f3546a4" gracePeriod=30 Oct 30 02:31:18.011887 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:18.011859 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft"] Oct 30 02:31:18.012071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:18.012038 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" podUID="f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" containerName="istio-proxy" containerID="cri-o://d6b4b15fa2b40704aff9c0cf6366a1d36fa746b3e2b401f9bac47d41b9e0ff5e" gracePeriod=30 Oct 30 02:31:18.900031 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:18.899991 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:19.138937 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:19.138896 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-57s4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:31:19.139157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:19.138952 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" podUID="caaec13d-e62d-4450-864c-d9e3fb7ca8d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:31:20.417751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:20.417716 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:20.747438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:20.747342 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:31:22.004772 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:22.004728 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:23.342471 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.342443 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:31:23.379964 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.379927 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 02:31:23.438317 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.438281 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istiod-ca-cert\") pod \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " Oct 30 02:31:23.438610 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.438358 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-credential-socket\") pod \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " Oct 30 02:31:23.438610 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.438390 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-workload-socket\") pod \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " Oct 30 02:31:23.438610 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.438471 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-workload-certs\") pod \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " Oct 30 02:31:23.438610 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.438519 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-data\") pod \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " Oct 30 02:31:23.438610 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.438569 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-envoy\") pod \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " Oct 30 02:31:23.438610 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.438599 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-envoy\") pod \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " Oct 30 02:31:23.438967 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.438646 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-kube-api-access-p2mj8\" (UniqueName: \"kubernetes.io/projected/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-kube-api-access-p2mj8\") pod \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " Oct 30 02:31:23.438967 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.438716 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-token\" (UniqueName: \"kubernetes.io/projected/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-token\") pod \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " Oct 30 02:31:23.438967 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.438749 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-workload-socket" pod "7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" (UID: "7478a4a2-ef1a-4d42-9c11-4ca61c11ac70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.438967 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.438789 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-credential-socket\") pod \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " Oct 30 02:31:23.438967 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.438879 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-workload-socket\") pod \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " Oct 30 02:31:23.438967 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.438935 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istiod-ca-cert\") pod \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.439011 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-podinfo\") pod \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.439055 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-kube-api-access-tqfr8\" (UniqueName: \"kubernetes.io/projected/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-kube-api-access-tqfr8\") pod \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.439084 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-data\") pod \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.439242 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-token\" (UniqueName: \"kubernetes.io/projected/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-token\") pod \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.439291 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-podinfo\") pod \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\" (UID: \"caaec13d-e62d-4450-864c-d9e3fb7ca8d4\") " Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.439347 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-workload-certs\") pod \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\" (UID: \"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70\") " Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.439339 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-credential-socket" pod "7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" (UID: "7478a4a2-ef1a-4d42-9c11-4ca61c11ac70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.440000 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-data" pod "caaec13d-e62d-4450-864c-d9e3fb7ca8d4" (UID: "caaec13d-e62d-4450-864c-d9e3fb7ca8d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.440207 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-workload-socket" pod "caaec13d-e62d-4450-864c-d9e3fb7ca8d4" (UID: "caaec13d-e62d-4450-864c-d9e3fb7ca8d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.440210 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-workload-certs" pod "7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" (UID: "7478a4a2-ef1a-4d42-9c11-4ca61c11ac70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.440415 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-data" pod "7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" (UID: "7478a4a2-ef1a-4d42-9c11-4ca61c11ac70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.440467 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istiod-ca-cert" pod "7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" (UID: "7478a4a2-ef1a-4d42-9c11-4ca61c11ac70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.440738 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.440759 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.440773 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.440781 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.440789 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.440797 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.440805 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.441172 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-credential-socket" pod "caaec13d-e62d-4450-864c-d9e3fb7ca8d4" (UID: "caaec13d-e62d-4450-864c-d9e3fb7ca8d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.441353 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-workload-certs" pod "caaec13d-e62d-4450-864c-d9e3fb7ca8d4" (UID: "caaec13d-e62d-4450-864c-d9e3fb7ca8d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.442382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.441957 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istiod-ca-cert" pod "caaec13d-e62d-4450-864c-d9e3fb7ca8d4" (UID: "caaec13d-e62d-4450-864c-d9e3fb7ca8d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:31:23.447010 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.446928 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-envoy" pod "7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" (UID: "7478a4a2-ef1a-4d42-9c11-4ca61c11ac70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.447792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.447626 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-token" pod "7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" (UID: "7478a4a2-ef1a-4d42-9c11-4ca61c11ac70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:31:23.447792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.447732 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-kube-api-access-tqfr8" pod "caaec13d-e62d-4450-864c-d9e3fb7ca8d4" (UID: "caaec13d-e62d-4450-864c-d9e3fb7ca8d4"). InnerVolumeSpecName "kube-api-access-tqfr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:31:23.447792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.447754 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-podinfo" pod "7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" (UID: "7478a4a2-ef1a-4d42-9c11-4ca61c11ac70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:31:23.448418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.448384 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-envoy" pod "caaec13d-e62d-4450-864c-d9e3fb7ca8d4" (UID: "caaec13d-e62d-4450-864c-d9e3fb7ca8d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.448898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.448869 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-token" pod "caaec13d-e62d-4450-864c-d9e3fb7ca8d4" (UID: "caaec13d-e62d-4450-864c-d9e3fb7ca8d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:31:23.449956 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.449214 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-podinfo" pod "caaec13d-e62d-4450-864c-d9e3fb7ca8d4" (UID: "caaec13d-e62d-4450-864c-d9e3fb7ca8d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:31:23.454501 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.454462 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-kube-api-access-p2mj8" pod "7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" (UID: "7478a4a2-ef1a-4d42-9c11-4ca61c11ac70"). InnerVolumeSpecName "kube-api-access-p2mj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:31:23.467648 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.467620 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:31:23.491686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.491653 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:31:23.541492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.541439 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-kube-api-access-8mx55\" (UniqueName: \"kubernetes.io/projected/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-kube-api-access-8mx55\") pod \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " Oct 30 02:31:23.541716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.541497 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-istio-envoy\") pod \"9782ca9d-bd87-415c-bad8-e080814048e7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " Oct 30 02:31:23.541716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.541535 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-credential-socket\") pod \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " Oct 30 02:31:23.541716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.541562 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9782ca9d-bd87-415c-bad8-e080814048e7-istio-token\" (UniqueName: \"kubernetes.io/projected/9782ca9d-bd87-415c-bad8-e080814048e7-istio-token\") pod \"9782ca9d-bd87-415c-bad8-e080814048e7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " Oct 30 02:31:23.541716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.541628 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-data\") pod \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " Oct 30 02:31:23.541716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.541666 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9782ca9d-bd87-415c-bad8-e080814048e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9782ca9d-bd87-415c-bad8-e080814048e7-istio-podinfo\") pod \"9782ca9d-bd87-415c-bad8-e080814048e7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " Oct 30 02:31:23.542561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.542169 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-envoy\") pod \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " Oct 30 02:31:23.542561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.542238 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-istio-data\") pod \"9782ca9d-bd87-415c-bad8-e080814048e7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " Oct 30 02:31:23.542561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.542292 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9782ca9d-bd87-415c-bad8-e080814048e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9782ca9d-bd87-415c-bad8-e080814048e7-istiod-ca-cert\") pod \"9782ca9d-bd87-415c-bad8-e080814048e7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " Oct 30 02:31:23.542561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.542367 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istiod-ca-cert\") pod \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " Oct 30 02:31:23.542561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.542408 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-podinfo\") pod \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " Oct 30 02:31:23.542561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.542446 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-credential-socket\") pod \"9782ca9d-bd87-415c-bad8-e080814048e7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " Oct 30 02:31:23.542561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.542504 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-workload-socket\") pod \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " Oct 30 02:31:23.542561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.542543 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9782ca9d-bd87-415c-bad8-e080814048e7-kube-api-access-4qb9v\" (UniqueName: \"kubernetes.io/projected/9782ca9d-bd87-415c-bad8-e080814048e7-kube-api-access-4qb9v\") pod \"9782ca9d-bd87-415c-bad8-e080814048e7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " Oct 30 02:31:23.543060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.542592 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-workload-certs\") pod \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " Oct 30 02:31:23.543060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.542631 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-workload-certs\") pod \"9782ca9d-bd87-415c-bad8-e080814048e7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " Oct 30 02:31:23.543060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.542693 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-token\" (UniqueName: \"kubernetes.io/projected/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-token\") pod \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\" (UID: \"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6\") " Oct 30 02:31:23.543060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.542752 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-workload-socket\") pod \"9782ca9d-bd87-415c-bad8-e080814048e7\" (UID: \"9782ca9d-bd87-415c-bad8-e080814048e7\") " Oct 30 02:31:23.544293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.543357 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.544293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.543373 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.544293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.543383 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqfr8\" (UniqueName: \"kubernetes.io/projected/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-kube-api-access-tqfr8\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.544293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.543396 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.544293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.543410 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.544293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.543422 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.544293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.543434 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.544293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.543510 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.544293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.543524 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2mj8\" (UniqueName: \"kubernetes.io/projected/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70-kube-api-access-p2mj8\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.544293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.543536 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.544293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.543574 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caaec13d-e62d-4450-864c-d9e3fb7ca8d4-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.544293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.543818 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-workload-socket" pod "9782ca9d-bd87-415c-bad8-e080814048e7" (UID: "9782ca9d-bd87-415c-bad8-e080814048e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.544979 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.544400 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9782ca9d-bd87-415c-bad8-e080814048e7-istiod-ca-cert" pod "9782ca9d-bd87-415c-bad8-e080814048e7" (UID: "9782ca9d-bd87-415c-bad8-e080814048e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:31:23.545826 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.545603 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istiod-ca-cert" pod "f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" (UID: "f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:31:23.546293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.545981 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-workload-certs" pod "f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" (UID: "f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.547333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.546731 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-istio-envoy" pod "9782ca9d-bd87-415c-bad8-e080814048e7" (UID: "9782ca9d-bd87-415c-bad8-e080814048e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.547333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.547004 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-credential-socket" pod "f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" (UID: "f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.547824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.547591 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-data" pod "f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" (UID: "f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.548295 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.547993 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-credential-socket" pod "9782ca9d-bd87-415c-bad8-e080814048e7" (UID: "9782ca9d-bd87-415c-bad8-e080814048e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.548524 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.548458 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-workload-certs" pod "9782ca9d-bd87-415c-bad8-e080814048e7" (UID: "9782ca9d-bd87-415c-bad8-e080814048e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.549062 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.549036 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-workload-socket" pod "f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" (UID: "f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.549470 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.549334 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-istio-data" pod "9782ca9d-bd87-415c-bad8-e080814048e7" (UID: "9782ca9d-bd87-415c-bad8-e080814048e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.551800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.551759 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9782ca9d-bd87-415c-bad8-e080814048e7-istio-podinfo" pod "9782ca9d-bd87-415c-bad8-e080814048e7" (UID: "9782ca9d-bd87-415c-bad8-e080814048e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:31:23.551800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.551773 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9782ca9d-bd87-415c-bad8-e080814048e7-istio-token" pod "9782ca9d-bd87-415c-bad8-e080814048e7" (UID: "9782ca9d-bd87-415c-bad8-e080814048e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:31:23.551971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.551816 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-kube-api-access-8mx55" pod "f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" (UID: "f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6"). InnerVolumeSpecName "kube-api-access-8mx55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:31:23.551971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.551911 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-podinfo" pod "f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" (UID: "f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:31:23.552167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.551977 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9782ca9d-bd87-415c-bad8-e080814048e7-kube-api-access-4qb9v" pod "9782ca9d-bd87-415c-bad8-e080814048e7" (UID: "9782ca9d-bd87-415c-bad8-e080814048e7"). InnerVolumeSpecName "kube-api-access-4qb9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:31:23.552167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.552047 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-envoy" pod "f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" (UID: "f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:31:23.553524 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.553497 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-token" pod "f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" (UID: "f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:31:23.645693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.645608 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.645693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.645634 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.645693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.645644 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mx55\" (UniqueName: \"kubernetes.io/projected/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-kube-api-access-8mx55\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.645693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.645654 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.645693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.645663 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.645693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.645672 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9782ca9d-bd87-415c-bad8-e080814048e7-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.645693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.645680 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.645693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.645688 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9782ca9d-bd87-415c-bad8-e080814048e7-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.645693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.645696 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.646146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.645703 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.646146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.645710 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9782ca9d-bd87-415c-bad8-e080814048e7-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.646146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.645719 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.646146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.645727 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.646146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.645737 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.646146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.645745 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.646146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.645753 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qb9v\" (UniqueName: \"kubernetes.io/projected/9782ca9d-bd87-415c-bad8-e080814048e7-kube-api-access-4qb9v\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.646146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.645761 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.646146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.645769 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9782ca9d-bd87-415c-bad8-e080814048e7-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:31:23.690307 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.690260 5998 generic.go:356] "Generic (PLEG): container finished" podID="caaec13d-e62d-4450-864c-d9e3fb7ca8d4" containerID="dd5450b72b56f3408b6dd3dd3799b0306b2253d06e54c4d3421bd5ecd1bcfd4e" exitCode=0 Oct 30 02:31:23.690503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.690324 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" Oct 30 02:31:23.690503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.690335 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" event={"ID":"caaec13d-e62d-4450-864c-d9e3fb7ca8d4","Type":"ContainerDied","Data":"dd5450b72b56f3408b6dd3dd3799b0306b2253d06e54c4d3421bd5ecd1bcfd4e"} Oct 30 02:31:23.690503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.690404 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v" event={"ID":"caaec13d-e62d-4450-864c-d9e3fb7ca8d4","Type":"ContainerDied","Data":"968fdf21f5532bd038957d8026bf32548f402eda004493265226863cfada2d7d"} Oct 30 02:31:23.690503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.690429 5998 scope.go:122] "RemoveContainer" containerID="dd5450b72b56f3408b6dd3dd3799b0306b2253d06e54c4d3421bd5ecd1bcfd4e" Oct 30 02:31:23.692464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.692437 5998 generic.go:356] "Generic (PLEG): container finished" podID="7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" containerID="84c0ff3b7967a3cfda918ac128b2b0c927359ef6f7a01a5a49fc77efdb625b73" exitCode=0 Oct 30 02:31:23.692603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.692478 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" event={"ID":"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70","Type":"ContainerDied","Data":"84c0ff3b7967a3cfda918ac128b2b0c927359ef6f7a01a5a49fc77efdb625b73"} Oct 30 02:31:23.692603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.692533 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" Oct 30 02:31:23.692603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.692547 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh" event={"ID":"7478a4a2-ef1a-4d42-9c11-4ca61c11ac70","Type":"ContainerDied","Data":"ea8cda631762817050b2f4606a8c28b62cf6f57a885bc6b6ccb044eceed0e636"} Oct 30 02:31:23.695549 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.695520 5998 generic.go:356] "Generic (PLEG): container finished" podID="f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" containerID="d6b4b15fa2b40704aff9c0cf6366a1d36fa746b3e2b401f9bac47d41b9e0ff5e" exitCode=0 Oct 30 02:31:23.695741 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.695725 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" Oct 30 02:31:23.695853 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.695826 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" event={"ID":"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6","Type":"ContainerDied","Data":"d6b4b15fa2b40704aff9c0cf6366a1d36fa746b3e2b401f9bac47d41b9e0ff5e"} Oct 30 02:31:23.695947 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.695878 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft" event={"ID":"f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6","Type":"ContainerDied","Data":"2d45619b585edc7253fdefd14779aadb9afbc887f093a000316ffef165c0e055"} Oct 30 02:31:23.701160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.700967 5998 generic.go:356] "Generic (PLEG): container finished" podID="9782ca9d-bd87-415c-bad8-e080814048e7" containerID="d0d222cc228a1afd865c05be868c3014eb31cfafa83a82c0e44d46117f3546a4" exitCode=0 Oct 30 02:31:23.701160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.701018 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" event={"ID":"9782ca9d-bd87-415c-bad8-e080814048e7","Type":"ContainerDied","Data":"d0d222cc228a1afd865c05be868c3014eb31cfafa83a82c0e44d46117f3546a4"} Oct 30 02:31:23.701160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.701055 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" event={"ID":"9782ca9d-bd87-415c-bad8-e080814048e7","Type":"ContainerDied","Data":"cc03c13477795081edcde2c16921ff762181e0fbf6dd7c11ee510eff2f9f8c56"} Oct 30 02:31:23.702562 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.701445 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7" Oct 30 02:31:23.710027 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.709500 5998 scope.go:122] "RemoveContainer" containerID="dd5450b72b56f3408b6dd3dd3799b0306b2253d06e54c4d3421bd5ecd1bcfd4e" Oct 30 02:31:23.710389 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:31:23.710359 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5450b72b56f3408b6dd3dd3799b0306b2253d06e54c4d3421bd5ecd1bcfd4e\": container with ID starting with dd5450b72b56f3408b6dd3dd3799b0306b2253d06e54c4d3421bd5ecd1bcfd4e not found: ID does not exist" containerID="dd5450b72b56f3408b6dd3dd3799b0306b2253d06e54c4d3421bd5ecd1bcfd4e" Oct 30 02:31:23.710570 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.710530 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5450b72b56f3408b6dd3dd3799b0306b2253d06e54c4d3421bd5ecd1bcfd4e"} err="failed to get container status \"dd5450b72b56f3408b6dd3dd3799b0306b2253d06e54c4d3421bd5ecd1bcfd4e\": rpc error: code = NotFound desc = could not find container \"dd5450b72b56f3408b6dd3dd3799b0306b2253d06e54c4d3421bd5ecd1bcfd4e\": container with ID starting with dd5450b72b56f3408b6dd3dd3799b0306b2253d06e54c4d3421bd5ecd1bcfd4e not found: ID does not exist" Oct 30 02:31:23.710570 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.710565 5998 scope.go:122] "RemoveContainer" containerID="84c0ff3b7967a3cfda918ac128b2b0c927359ef6f7a01a5a49fc77efdb625b73" Oct 30 02:31:23.723424 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.723396 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:23.725502 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.725478 5998 scope.go:122] "RemoveContainer" containerID="84c0ff3b7967a3cfda918ac128b2b0c927359ef6f7a01a5a49fc77efdb625b73" Oct 30 02:31:23.725769 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:31:23.725746 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84c0ff3b7967a3cfda918ac128b2b0c927359ef6f7a01a5a49fc77efdb625b73\": container with ID starting with 84c0ff3b7967a3cfda918ac128b2b0c927359ef6f7a01a5a49fc77efdb625b73 not found: ID does not exist" containerID="84c0ff3b7967a3cfda918ac128b2b0c927359ef6f7a01a5a49fc77efdb625b73" Oct 30 02:31:23.725822 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.725777 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84c0ff3b7967a3cfda918ac128b2b0c927359ef6f7a01a5a49fc77efdb625b73"} err="failed to get container status \"84c0ff3b7967a3cfda918ac128b2b0c927359ef6f7a01a5a49fc77efdb625b73\": rpc error: code = NotFound desc = could not find container \"84c0ff3b7967a3cfda918ac128b2b0c927359ef6f7a01a5a49fc77efdb625b73\": container with ID starting with 84c0ff3b7967a3cfda918ac128b2b0c927359ef6f7a01a5a49fc77efdb625b73 not found: ID does not exist" Oct 30 02:31:23.725822 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.725795 5998 scope.go:122] "RemoveContainer" containerID="d6b4b15fa2b40704aff9c0cf6366a1d36fa746b3e2b401f9bac47d41b9e0ff5e" Oct 30 02:31:23.735961 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.735943 5998 scope.go:122] "RemoveContainer" containerID="d6b4b15fa2b40704aff9c0cf6366a1d36fa746b3e2b401f9bac47d41b9e0ff5e" Oct 30 02:31:23.736359 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:31:23.736336 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b4b15fa2b40704aff9c0cf6366a1d36fa746b3e2b401f9bac47d41b9e0ff5e\": container with ID starting with d6b4b15fa2b40704aff9c0cf6366a1d36fa746b3e2b401f9bac47d41b9e0ff5e not found: ID does not exist" containerID="d6b4b15fa2b40704aff9c0cf6366a1d36fa746b3e2b401f9bac47d41b9e0ff5e" Oct 30 02:31:23.736430 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.736366 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b4b15fa2b40704aff9c0cf6366a1d36fa746b3e2b401f9bac47d41b9e0ff5e"} err="failed to get container status \"d6b4b15fa2b40704aff9c0cf6366a1d36fa746b3e2b401f9bac47d41b9e0ff5e\": rpc error: code = NotFound desc = could not find container \"d6b4b15fa2b40704aff9c0cf6366a1d36fa746b3e2b401f9bac47d41b9e0ff5e\": container with ID starting with d6b4b15fa2b40704aff9c0cf6366a1d36fa746b3e2b401f9bac47d41b9e0ff5e not found: ID does not exist" Oct 30 02:31:23.736430 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.736385 5998 scope.go:122] "RemoveContainer" containerID="d0d222cc228a1afd865c05be868c3014eb31cfafa83a82c0e44d46117f3546a4" Oct 30 02:31:23.746501 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.746483 5998 scope.go:122] "RemoveContainer" containerID="d0d222cc228a1afd865c05be868c3014eb31cfafa83a82c0e44d46117f3546a4" Oct 30 02:31:23.746769 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:31:23.746742 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0d222cc228a1afd865c05be868c3014eb31cfafa83a82c0e44d46117f3546a4\": container with ID starting with d0d222cc228a1afd865c05be868c3014eb31cfafa83a82c0e44d46117f3546a4 not found: ID does not exist" containerID="d0d222cc228a1afd865c05be868c3014eb31cfafa83a82c0e44d46117f3546a4" Oct 30 02:31:23.746849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.746779 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0d222cc228a1afd865c05be868c3014eb31cfafa83a82c0e44d46117f3546a4"} err="failed to get container status \"d0d222cc228a1afd865c05be868c3014eb31cfafa83a82c0e44d46117f3546a4\": rpc error: code = NotFound desc = could not find container \"d0d222cc228a1afd865c05be868c3014eb31cfafa83a82c0e44d46117f3546a4\": container with ID starting with d0d222cc228a1afd865c05be868c3014eb31cfafa83a82c0e44d46117f3546a4 not found: ID does not exist" Oct 30 02:31:23.791930 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.791862 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh"] Oct 30 02:31:23.796343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.796311 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ssskh"] Oct 30 02:31:23.867903 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.865039 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:23.877779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.877744 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7"] Oct 30 02:31:23.881531 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.881499 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fpsp7"] Oct 30 02:31:23.890716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.890679 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v"] Oct 30 02:31:23.894670 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.894638 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-57s4v"] Oct 30 02:31:23.923849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.923823 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft"] Oct 30 02:31:23.950022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:23.949989 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-jhqft"] Oct 30 02:31:24.012276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:24.012243 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:24.541978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:24.541938 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" path="/var/lib/kubelet/pods/7478a4a2-ef1a-4d42-9c11-4ca61c11ac70/volumes" Oct 30 02:31:24.542923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:24.542898 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9782ca9d-bd87-415c-bad8-e080814048e7" path="/var/lib/kubelet/pods/9782ca9d-bd87-415c-bad8-e080814048e7/volumes" Oct 30 02:31:24.543490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:24.543470 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caaec13d-e62d-4450-864c-d9e3fb7ca8d4" path="/var/lib/kubelet/pods/caaec13d-e62d-4450-864c-d9e3fb7ca8d4/volumes" Oct 30 02:31:24.544014 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:24.543995 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" path="/var/lib/kubelet/pods/f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6/volumes" Oct 30 02:31:25.220243 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:31:25.220198 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:31:25.548549 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:25.547591 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:27.322274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:27.322235 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:28.884165 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:28.884127 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:30.477889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:30.477849 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:31.992189 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:31.992081 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:33.569234 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:33.569197 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:34.067983 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:34.067947 5998 scope.go:122] "RemoveContainer" containerID="a69190d8cd0c0bc2b4da998d4f86bdf46360c5641ba82c1d77de5c2896f04d9e" Oct 30 02:31:34.078757 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:34.078732 5998 scope.go:122] "RemoveContainer" containerID="cb8d69a52deb1cd91ccd64b79ed4ee287e9d5181718708f266d4236b26a4de31" Oct 30 02:31:34.088945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:34.088918 5998 scope.go:122] "RemoveContainer" containerID="8c2423a97b13e463848db9518cc2dc9fb4c157c3406355025b69677720442d71" Oct 30 02:31:34.098533 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:34.098508 5998 scope.go:122] "RemoveContainer" containerID="70df15320129a966040eb602df406a26903b0d4ccdb8f4c817ecd8923cf2a129" Oct 30 02:31:35.173585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:35.173547 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:36.782692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:36.782651 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:38.383103 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:38.383048 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:39.701672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:39.701632 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:39.990074 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:39.989981 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:41.485783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:41.485745 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:43.080073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:43.080040 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:44.594786 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:44.594745 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:46.280989 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:46.280950 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:47.882824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:47.882792 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:49.410323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:49.410280 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:50.987038 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:50.987003 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:52.510491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:52.510445 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:54.114970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:54.114930 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:54.710587 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:54.710544 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl"] Oct 30 02:31:54.710843 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:54.710814 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" podUID="1411791d-2b8f-46e0-8b26-8799563d8c4b" containerName="istio-proxy" containerID="cri-o://636c8cafb70c3d354c5985475af955149a1902504a3990d4722f7e3115f75136" gracePeriod=30 Oct 30 02:31:55.223818 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:31:55.223775 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:31:55.675396 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:55.675360 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:57.207006 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:57.206966 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:58.807522 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:58.807477 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:31:59.689303 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:59.689260 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:31:59.689484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:59.689331 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" podUID="1411791d-2b8f-46e0-8b26-8799563d8c4b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:31:59.999876 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:31:59.999849 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:32:00.020919 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.020886 5998 generic.go:356] "Generic (PLEG): container finished" podID="1411791d-2b8f-46e0-8b26-8799563d8c4b" containerID="636c8cafb70c3d354c5985475af955149a1902504a3990d4722f7e3115f75136" exitCode=0 Oct 30 02:32:00.021133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.020944 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" Oct 30 02:32:00.021133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.020948 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" event={"ID":"1411791d-2b8f-46e0-8b26-8799563d8c4b","Type":"ContainerDied","Data":"636c8cafb70c3d354c5985475af955149a1902504a3990d4722f7e3115f75136"} Oct 30 02:32:00.021133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.021082 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl" event={"ID":"1411791d-2b8f-46e0-8b26-8799563d8c4b","Type":"ContainerDied","Data":"f45237dcab745d06976482bae8504ede03c34ccb1e9898d3602004d8d35b3203"} Oct 30 02:32:00.021133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.021112 5998 scope.go:122] "RemoveContainer" containerID="636c8cafb70c3d354c5985475af955149a1902504a3990d4722f7e3115f75136" Oct 30 02:32:00.032432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.032404 5998 scope.go:122] "RemoveContainer" containerID="c0b725d9ba4c863b51985264780bce7ae018b22fcf1c74e2af9608c380d8d1da" Oct 30 02:32:00.042927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.042887 5998 scope.go:122] "RemoveContainer" containerID="636c8cafb70c3d354c5985475af955149a1902504a3990d4722f7e3115f75136" Oct 30 02:32:00.043278 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:32:00.043249 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"636c8cafb70c3d354c5985475af955149a1902504a3990d4722f7e3115f75136\": container with ID starting with 636c8cafb70c3d354c5985475af955149a1902504a3990d4722f7e3115f75136 not found: ID does not exist" containerID="636c8cafb70c3d354c5985475af955149a1902504a3990d4722f7e3115f75136" Oct 30 02:32:00.043363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.043296 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"636c8cafb70c3d354c5985475af955149a1902504a3990d4722f7e3115f75136"} err="failed to get container status \"636c8cafb70c3d354c5985475af955149a1902504a3990d4722f7e3115f75136\": rpc error: code = NotFound desc = could not find container \"636c8cafb70c3d354c5985475af955149a1902504a3990d4722f7e3115f75136\": container with ID starting with 636c8cafb70c3d354c5985475af955149a1902504a3990d4722f7e3115f75136 not found: ID does not exist" Oct 30 02:32:00.043363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.043323 5998 scope.go:122] "RemoveContainer" containerID="c0b725d9ba4c863b51985264780bce7ae018b22fcf1c74e2af9608c380d8d1da" Oct 30 02:32:00.043650 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:32:00.043621 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b725d9ba4c863b51985264780bce7ae018b22fcf1c74e2af9608c380d8d1da\": container with ID starting with c0b725d9ba4c863b51985264780bce7ae018b22fcf1c74e2af9608c380d8d1da not found: ID does not exist" containerID="c0b725d9ba4c863b51985264780bce7ae018b22fcf1c74e2af9608c380d8d1da" Oct 30 02:32:00.043715 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.043665 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b725d9ba4c863b51985264780bce7ae018b22fcf1c74e2af9608c380d8d1da"} err="failed to get container status \"c0b725d9ba4c863b51985264780bce7ae018b22fcf1c74e2af9608c380d8d1da\": rpc error: code = NotFound desc = could not find container \"c0b725d9ba4c863b51985264780bce7ae018b22fcf1c74e2af9608c380d8d1da\": container with ID starting with c0b725d9ba4c863b51985264780bce7ae018b22fcf1c74e2af9608c380d8d1da not found: ID does not exist" Oct 30 02:32:00.122433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.122391 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1411791d-2b8f-46e0-8b26-8799563d8c4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1411791d-2b8f-46e0-8b26-8799563d8c4b-istiod-ca-cert\") pod \"1411791d-2b8f-46e0-8b26-8799563d8c4b\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " Oct 30 02:32:00.122630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.122452 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-token\" (UniqueName: \"kubernetes.io/projected/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-token\") pod \"1411791d-2b8f-46e0-8b26-8799563d8c4b\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " Oct 30 02:32:00.122630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.122506 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-podinfo\") pod \"1411791d-2b8f-46e0-8b26-8799563d8c4b\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " Oct 30 02:32:00.122630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.122601 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1411791d-2b8f-46e0-8b26-8799563d8c4b-kube-api-access-drtrv\" (UniqueName: \"kubernetes.io/projected/1411791d-2b8f-46e0-8b26-8799563d8c4b-kube-api-access-drtrv\") pod \"1411791d-2b8f-46e0-8b26-8799563d8c4b\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " Oct 30 02:32:00.122808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.122642 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-workload-socket\") pod \"1411791d-2b8f-46e0-8b26-8799563d8c4b\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " Oct 30 02:32:00.122808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.122687 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-workload-certs\") pod \"1411791d-2b8f-46e0-8b26-8799563d8c4b\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " Oct 30 02:32:00.122808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.122722 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-data\") pod \"1411791d-2b8f-46e0-8b26-8799563d8c4b\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " Oct 30 02:32:00.122982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.122833 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-envoy\") pod \"1411791d-2b8f-46e0-8b26-8799563d8c4b\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " Oct 30 02:32:00.122982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.122845 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1411791d-2b8f-46e0-8b26-8799563d8c4b-istiod-ca-cert" pod "1411791d-2b8f-46e0-8b26-8799563d8c4b" (UID: "1411791d-2b8f-46e0-8b26-8799563d8c4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:32:00.123130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.123009 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-workload-certs" pod "1411791d-2b8f-46e0-8b26-8799563d8c4b" (UID: "1411791d-2b8f-46e0-8b26-8799563d8c4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:32:00.123130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.123034 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-credential-socket\") pod \"1411791d-2b8f-46e0-8b26-8799563d8c4b\" (UID: \"1411791d-2b8f-46e0-8b26-8799563d8c4b\") " Oct 30 02:32:00.123252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.123154 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-data" pod "1411791d-2b8f-46e0-8b26-8799563d8c4b" (UID: "1411791d-2b8f-46e0-8b26-8799563d8c4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:32:00.123489 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.123453 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-workload-socket" pod "1411791d-2b8f-46e0-8b26-8799563d8c4b" (UID: "1411791d-2b8f-46e0-8b26-8799563d8c4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:32:00.123959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.123687 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-credential-socket" pod "1411791d-2b8f-46e0-8b26-8799563d8c4b" (UID: "1411791d-2b8f-46e0-8b26-8799563d8c4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:32:00.123959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.123861 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:32:00.123959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.123881 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1411791d-2b8f-46e0-8b26-8799563d8c4b-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:32:00.123959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.123897 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:32:00.123959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.123909 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:32:00.123959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.123922 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:32:00.125492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.125470 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1411791d-2b8f-46e0-8b26-8799563d8c4b-kube-api-access-drtrv" pod "1411791d-2b8f-46e0-8b26-8799563d8c4b" (UID: "1411791d-2b8f-46e0-8b26-8799563d8c4b"). InnerVolumeSpecName "kube-api-access-drtrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:32:00.125723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.125704 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-podinfo" pod "1411791d-2b8f-46e0-8b26-8799563d8c4b" (UID: "1411791d-2b8f-46e0-8b26-8799563d8c4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:32:00.126661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.126636 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-envoy" pod "1411791d-2b8f-46e0-8b26-8799563d8c4b" (UID: "1411791d-2b8f-46e0-8b26-8799563d8c4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:32:00.126808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.126791 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-token" pod "1411791d-2b8f-46e0-8b26-8799563d8c4b" (UID: "1411791d-2b8f-46e0-8b26-8799563d8c4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:32:00.225351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.225249 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:32:00.225351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.225285 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:32:00.225351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.225299 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drtrv\" (UniqueName: \"kubernetes.io/projected/1411791d-2b8f-46e0-8b26-8799563d8c4b-kube-api-access-drtrv\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:32:00.225351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.225313 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1411791d-2b8f-46e0-8b26-8799563d8c4b-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:32:00.367489 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.367453 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl"] Oct 30 02:32:00.377739 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.377704 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-58fad4-openshift-default-7d558dc9c7-888fl"] Oct 30 02:32:00.395353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.395316 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:00.540932 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:00.540835 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1411791d-2b8f-46e0-8b26-8799563d8c4b" path="/var/lib/kubelet/pods/1411791d-2b8f-46e0-8b26-8799563d8c4b/volumes" Oct 30 02:32:01.989336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:01.989296 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:03.672741 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:03.672703 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:05.388103 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:05.388063 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:06.918923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:06.918886 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:08.600046 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:08.599965 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:10.594619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:10.594571 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:12.200886 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:12.200851 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:13.912580 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:13.912535 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:15.520395 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:15.520350 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:17.062781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:17.062742 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:18.910103 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:18.910052 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:19.037045 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:19.037007 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7"] Oct 30 02:32:19.046561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:19.046526 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3507eb0-qwlf7"] Oct 30 02:32:20.459362 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:20.459320 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:20.558008 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:20.557970 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae0f38fe-b272-4916-89a3-7dcf32ea18d6" path="/var/lib/kubelet/pods/ae0f38fe-b272-4916-89a3-7dcf32ea18d6/volumes" Oct 30 02:32:22.094629 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:22.094563 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:23.750306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:23.741546 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:24.853170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:24.853136 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:25.250645 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:32:25.250566 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:32:25.527332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:25.527238 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:27.321822 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.321761 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:27.530111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.529306 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv"] Oct 30 02:32:27.530339 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.530296 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee" containerName="cleanup" Oct 30 02:32:27.530339 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.530318 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee" containerName="cleanup" Oct 30 02:32:27.530339 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.530335 5998 cpu_manager.go:414] "containerMap: removing container" podUID="e5882ea9-fe2e-4587-abb7-bb14939f53f6" containerName="cleanup" Oct 30 02:32:27.530543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.530343 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5882ea9-fe2e-4587-abb7-bb14939f53f6" containerName="cleanup" Oct 30 02:32:27.530543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.530360 5998 cpu_manager.go:414] "containerMap: removing container" podUID="2f3ead60-88ca-42af-9e25-7822589058e4" containerName="cleanup" Oct 30 02:32:27.530543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.530374 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3ead60-88ca-42af-9e25-7822589058e4" containerName="cleanup" Oct 30 02:32:27.530543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.530386 5998 cpu_manager.go:414] "containerMap: removing container" podUID="96bc88e1-7b35-4b54-b12f-69227ae41a88" containerName="cleanup" Oct 30 02:32:27.530543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.530393 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96bc88e1-7b35-4b54-b12f-69227ae41a88" containerName="cleanup" Oct 30 02:32:27.530543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.530406 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c9b0ea41-0e61-456d-bbce-6ee7145552ea" containerName="cleanup" Oct 30 02:32:27.530543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.530413 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b0ea41-0e61-456d-bbce-6ee7145552ea" containerName="cleanup" Oct 30 02:32:27.530543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.530424 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8e81c096-2704-4d9e-80b8-9b1f05e7151f" containerName="cleanup" Oct 30 02:32:27.530543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.530431 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e81c096-2704-4d9e-80b8-9b1f05e7151f" containerName="cleanup" Oct 30 02:32:27.530543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.530447 5998 cpu_manager.go:414] "containerMap: removing container" podUID="860ba768-1529-403e-ba81-13395d6cee83" containerName="cleanup" Oct 30 02:32:27.530543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.530456 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860ba768-1529-403e-ba81-13395d6cee83" containerName="cleanup" Oct 30 02:32:27.530543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.530468 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb" containerName="cleanup" Oct 30 02:32:27.530543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.530475 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.530484 5998 cpu_manager.go:414] "containerMap: removing container" podUID="115cd7ff-80aa-4cd9-a247-9154fe7d117f" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537230 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115cd7ff-80aa-4cd9-a247-9154fe7d117f" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537246 5998 cpu_manager.go:414] "containerMap: removing container" podUID="860ba768-1529-403e-ba81-13395d6cee83" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537258 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860ba768-1529-403e-ba81-13395d6cee83" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537278 5998 cpu_manager.go:414] "containerMap: removing container" podUID="046ebf9b-3d7e-4c67-bfcf-8545980b59ff" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537286 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="046ebf9b-3d7e-4c67-bfcf-8545980b59ff" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537297 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c9b0ea41-0e61-456d-bbce-6ee7145552ea" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537304 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b0ea41-0e61-456d-bbce-6ee7145552ea" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537332 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7ff76844-137e-44ec-b146-4d9c84997ada" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537342 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff76844-137e-44ec-b146-4d9c84997ada" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537352 5998 cpu_manager.go:414] "containerMap: removing container" podUID="5bce6d68-142e-46ef-b408-89e44ca0fc10" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537359 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bce6d68-142e-46ef-b408-89e44ca0fc10" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537371 5998 cpu_manager.go:414] "containerMap: removing container" podUID="0771e950-74b9-45cb-a7be-ccb30c8c8c65" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537378 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0771e950-74b9-45cb-a7be-ccb30c8c8c65" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537389 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7ff76844-137e-44ec-b146-4d9c84997ada" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537397 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff76844-137e-44ec-b146-4d9c84997ada" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537409 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3f69dd9b-8eda-4368-b693-33c5b9721710" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537417 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f69dd9b-8eda-4368-b693-33c5b9721710" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537430 5998 cpu_manager.go:414] "containerMap: removing container" podUID="512be8ad-51e7-4919-a456-9fcfafab46bd" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537437 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512be8ad-51e7-4919-a456-9fcfafab46bd" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537449 5998 cpu_manager.go:414] "containerMap: removing container" podUID="daaa2b88-a49a-4a88-8325-33307e969074" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537456 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daaa2b88-a49a-4a88-8325-33307e969074" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537469 5998 cpu_manager.go:414] "containerMap: removing container" podUID="01d4f72c-dc27-4988-b301-873c276855bd" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537476 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d4f72c-dc27-4988-b301-873c276855bd" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537487 5998 cpu_manager.go:414] "containerMap: removing container" podUID="0771e950-74b9-45cb-a7be-ccb30c8c8c65" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537494 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0771e950-74b9-45cb-a7be-ccb30c8c8c65" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537510 5998 cpu_manager.go:414] "containerMap: removing container" podUID="f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" containerName="istio-proxy" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537517 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" containerName="istio-proxy" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537530 5998 cpu_manager.go:414] "containerMap: removing container" podUID="860ba768-1529-403e-ba81-13395d6cee83" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537537 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860ba768-1529-403e-ba81-13395d6cee83" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537550 5998 cpu_manager.go:414] "containerMap: removing container" podUID="be309ab3-73d7-42c0-92a8-0fc686504795" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537558 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be309ab3-73d7-42c0-92a8-0fc686504795" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537570 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a105635a-7115-4c02-aa79-60516391579b" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537577 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a105635a-7115-4c02-aa79-60516391579b" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537586 5998 cpu_manager.go:414] "containerMap: removing container" podUID="9e4909d0-f3ae-423a-a478-4ee5ce9734a1" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537593 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e4909d0-f3ae-423a-a478-4ee5ce9734a1" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537605 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a1f42aec-2169-46f1-a837-039b920ac828" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537611 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1f42aec-2169-46f1-a837-039b920ac828" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537626 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1411791d-2b8f-46e0-8b26-8799563d8c4b" containerName="istio-proxy" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537633 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1411791d-2b8f-46e0-8b26-8799563d8c4b" containerName="istio-proxy" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537643 5998 cpu_manager.go:414] "containerMap: removing container" podUID="046ebf9b-3d7e-4c67-bfcf-8545980b59ff" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537650 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="046ebf9b-3d7e-4c67-bfcf-8545980b59ff" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537662 5998 cpu_manager.go:414] "containerMap: removing container" podUID="5bce6d68-142e-46ef-b408-89e44ca0fc10" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537669 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bce6d68-142e-46ef-b408-89e44ca0fc10" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537688 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a105635a-7115-4c02-aa79-60516391579b" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537695 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a105635a-7115-4c02-aa79-60516391579b" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537704 5998 cpu_manager.go:414] "containerMap: removing container" podUID="daaa2b88-a49a-4a88-8325-33307e969074" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537710 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daaa2b88-a49a-4a88-8325-33307e969074" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537720 5998 cpu_manager.go:414] "containerMap: removing container" podUID="6df3039d-5102-404f-b58f-eae1e79feb36" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537727 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df3039d-5102-404f-b58f-eae1e79feb36" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537747 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a1f42aec-2169-46f1-a837-039b920ac828" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537754 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1f42aec-2169-46f1-a837-039b920ac828" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537765 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3f69dd9b-8eda-4368-b693-33c5b9721710" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537773 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f69dd9b-8eda-4368-b693-33c5b9721710" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537791 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3f69dd9b-8eda-4368-b693-33c5b9721710" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537798 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f69dd9b-8eda-4368-b693-33c5b9721710" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537815 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537823 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537831 5998 cpu_manager.go:414] "containerMap: removing container" podUID="5bce6d68-142e-46ef-b408-89e44ca0fc10" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537839 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bce6d68-142e-46ef-b408-89e44ca0fc10" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537855 5998 cpu_manager.go:414] "containerMap: removing container" podUID="21de7884-5d25-40b9-9cc4-5f9f87eff813" containerName="discovery" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537863 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21de7884-5d25-40b9-9cc4-5f9f87eff813" containerName="discovery" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537875 5998 cpu_manager.go:414] "containerMap: removing container" podUID="caaec13d-e62d-4450-864c-d9e3fb7ca8d4" containerName="istio-proxy" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537882 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caaec13d-e62d-4450-864c-d9e3fb7ca8d4" containerName="istio-proxy" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537900 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537907 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537918 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a105635a-7115-4c02-aa79-60516391579b" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537925 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a105635a-7115-4c02-aa79-60516391579b" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537935 5998 cpu_manager.go:414] "containerMap: removing container" podUID="01d4f72c-dc27-4988-b301-873c276855bd" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537942 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d4f72c-dc27-4988-b301-873c276855bd" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537953 5998 cpu_manager.go:414] "containerMap: removing container" podUID="2f3ead60-88ca-42af-9e25-7822589058e4" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537960 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3ead60-88ca-42af-9e25-7822589058e4" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537972 5998 cpu_manager.go:414] "containerMap: removing container" podUID="9782ca9d-bd87-415c-bad8-e080814048e7" containerName="istio-proxy" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537979 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9782ca9d-bd87-415c-bad8-e080814048e7" containerName="istio-proxy" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.537993 5998 cpu_manager.go:414] "containerMap: removing container" podUID="daaa2b88-a49a-4a88-8325-33307e969074" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538000 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daaa2b88-a49a-4a88-8325-33307e969074" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538011 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1411791d-2b8f-46e0-8b26-8799563d8c4b" containerName="istio-proxy" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538019 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1411791d-2b8f-46e0-8b26-8799563d8c4b" containerName="istio-proxy" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538030 5998 cpu_manager.go:414] "containerMap: removing container" podUID="115cd7ff-80aa-4cd9-a247-9154fe7d117f" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538037 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115cd7ff-80aa-4cd9-a247-9154fe7d117f" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538048 5998 cpu_manager.go:414] "containerMap: removing container" podUID="512d72f9-4883-4d39-9df8-9282f56da2ac" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538056 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512d72f9-4883-4d39-9df8-9282f56da2ac" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538066 5998 cpu_manager.go:414] "containerMap: removing container" podUID="115cd7ff-80aa-4cd9-a247-9154fe7d117f" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538073 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115cd7ff-80aa-4cd9-a247-9154fe7d117f" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538106 5998 cpu_manager.go:414] "containerMap: removing container" podUID="512d72f9-4883-4d39-9df8-9282f56da2ac" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538113 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512d72f9-4883-4d39-9df8-9282f56da2ac" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538126 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" containerName="istio-proxy" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538133 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" containerName="istio-proxy" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538144 5998 cpu_manager.go:414] "containerMap: removing container" podUID="6df3039d-5102-404f-b58f-eae1e79feb36" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538152 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df3039d-5102-404f-b58f-eae1e79feb36" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538160 5998 cpu_manager.go:414] "containerMap: removing container" podUID="512d72f9-4883-4d39-9df8-9282f56da2ac" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538168 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512d72f9-4883-4d39-9df8-9282f56da2ac" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538179 5998 cpu_manager.go:414] "containerMap: removing container" podUID="01d4f72c-dc27-4988-b301-873c276855bd" containerName="cleanup" Oct 30 02:32:27.538113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538186 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01d4f72c-dc27-4988-b301-873c276855bd" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538205 5998 cpu_manager.go:414] "containerMap: removing container" podUID="96bc88e1-7b35-4b54-b12f-69227ae41a88" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538213 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96bc88e1-7b35-4b54-b12f-69227ae41a88" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538223 5998 cpu_manager.go:414] "containerMap: removing container" podUID="6df3039d-5102-404f-b58f-eae1e79feb36" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538230 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df3039d-5102-404f-b58f-eae1e79feb36" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538242 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8e81c096-2704-4d9e-80b8-9b1f05e7151f" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538249 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e81c096-2704-4d9e-80b8-9b1f05e7151f" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538258 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538265 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538276 5998 cpu_manager.go:414] "containerMap: removing container" podUID="9e4909d0-f3ae-423a-a478-4ee5ce9734a1" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538283 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e4909d0-f3ae-423a-a478-4ee5ce9734a1" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538293 5998 cpu_manager.go:414] "containerMap: removing container" podUID="e5882ea9-fe2e-4587-abb7-bb14939f53f6" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538300 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5882ea9-fe2e-4587-abb7-bb14939f53f6" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538313 5998 cpu_manager.go:414] "containerMap: removing container" podUID="512be8ad-51e7-4919-a456-9fcfafab46bd" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538320 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512be8ad-51e7-4919-a456-9fcfafab46bd" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538330 5998 cpu_manager.go:414] "containerMap: removing container" podUID="be309ab3-73d7-42c0-92a8-0fc686504795" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538337 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be309ab3-73d7-42c0-92a8-0fc686504795" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538349 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a1f42aec-2169-46f1-a837-039b920ac828" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538357 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1f42aec-2169-46f1-a837-039b920ac828" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538560 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538573 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a105635a-7115-4c02-aa79-60516391579b" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538581 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d4f72c-dc27-4988-b301-873c276855bd" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538590 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3ead60-88ca-42af-9e25-7822589058e4" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538601 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="9782ca9d-bd87-415c-bad8-e080814048e7" containerName="istio-proxy" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538611 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="daaa2b88-a49a-4a88-8325-33307e969074" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538620 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="512d72f9-4883-4d39-9df8-9282f56da2ac" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538630 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1411791d-2b8f-46e0-8b26-8799563d8c4b" containerName="istio-proxy" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538639 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="115cd7ff-80aa-4cd9-a247-9154fe7d117f" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538647 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="512d72f9-4883-4d39-9df8-9282f56da2ac" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538655 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="115cd7ff-80aa-4cd9-a247-9154fe7d117f" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538665 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e4909d0-f3ae-423a-a478-4ee5ce9734a1" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538675 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="512d72f9-4883-4d39-9df8-9282f56da2ac" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538685 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7478a4a2-ef1a-4d42-9c11-4ca61c11ac70" containerName="istio-proxy" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538697 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="be309ab3-73d7-42c0-92a8-0fc686504795" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538706 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df3039d-5102-404f-b58f-eae1e79feb36" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538715 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d4f72c-dc27-4988-b301-873c276855bd" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538725 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="96bc88e1-7b35-4b54-b12f-69227ae41a88" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538736 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df3039d-5102-404f-b58f-eae1e79feb36" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538743 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e81c096-2704-4d9e-80b8-9b1f05e7151f" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538751 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538761 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e4909d0-f3ae-423a-a478-4ee5ce9734a1" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538769 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5882ea9-fe2e-4587-abb7-bb14939f53f6" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538777 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="512be8ad-51e7-4919-a456-9fcfafab46bd" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538785 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="be309ab3-73d7-42c0-92a8-0fc686504795" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538795 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1f42aec-2169-46f1-a837-039b920ac828" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538803 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538810 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5882ea9-fe2e-4587-abb7-bb14939f53f6" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538822 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3ead60-88ca-42af-9e25-7822589058e4" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538833 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="96bc88e1-7b35-4b54-b12f-69227ae41a88" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538842 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b0ea41-0e61-456d-bbce-6ee7145552ea" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538852 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="860ba768-1529-403e-ba81-13395d6cee83" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538862 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1c2e04-ca3d-4cf6-b7ed-62b355a70dcb" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538873 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="115cd7ff-80aa-4cd9-a247-9154fe7d117f" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538881 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="860ba768-1529-403e-ba81-13395d6cee83" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538888 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="046ebf9b-3d7e-4c67-bfcf-8545980b59ff" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538895 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b0ea41-0e61-456d-bbce-6ee7145552ea" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538906 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff76844-137e-44ec-b146-4d9c84997ada" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538914 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bce6d68-142e-46ef-b408-89e44ca0fc10" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538923 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="0771e950-74b9-45cb-a7be-ccb30c8c8c65" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538930 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff76844-137e-44ec-b146-4d9c84997ada" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538940 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f69dd9b-8eda-4368-b693-33c5b9721710" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538949 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="512be8ad-51e7-4919-a456-9fcfafab46bd" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538958 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="01d4f72c-dc27-4988-b301-873c276855bd" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538969 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="0771e950-74b9-45cb-a7be-ccb30c8c8c65" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538978 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="f186c2e9-3b5d-48e7-b649-5bcd5cd38ce6" containerName="istio-proxy" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538990 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="860ba768-1529-403e-ba81-13395d6cee83" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.538998 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="be309ab3-73d7-42c0-92a8-0fc686504795" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.539007 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a105635a-7115-4c02-aa79-60516391579b" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.539016 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="96bc88e1-7b35-4b54-b12f-69227ae41a88" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.539024 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e4909d0-f3ae-423a-a478-4ee5ce9734a1" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.539032 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1f42aec-2169-46f1-a837-039b920ac828" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.539044 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bce6d68-142e-46ef-b408-89e44ca0fc10" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.539055 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a105635a-7115-4c02-aa79-60516391579b" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.539064 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="daaa2b88-a49a-4a88-8325-33307e969074" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.539072 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df3039d-5102-404f-b58f-eae1e79feb36" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.539082 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="046ebf9b-3d7e-4c67-bfcf-8545980b59ff" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.539108 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1f42aec-2169-46f1-a837-039b920ac828" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.539119 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e81c096-2704-4d9e-80b8-9b1f05e7151f" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.539128 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b0ea41-0e61-456d-bbce-6ee7145552ea" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.539139 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f69dd9b-8eda-4368-b693-33c5b9721710" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.539147 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="0771e950-74b9-45cb-a7be-ccb30c8c8c65" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.539157 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.539166 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bce6d68-142e-46ef-b408-89e44ca0fc10" containerName="cleanup" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.539176 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="21de7884-5d25-40b9-9cc4-5f9f87eff813" containerName="discovery" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.539185 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="caaec13d-e62d-4450-864c-d9e3fb7ca8d4" containerName="istio-proxy" Oct 30 02:32:27.549534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.547031 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:27.566646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.563542 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv"] Oct 30 02:32:27.567641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.567473 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-52d131\"/\"default-dockercfg-27j8n\"" Oct 30 02:32:27.567641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.567590 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-52d131\"/\"test-model-52d131-kserve-self-signed-certs\"" Oct 30 02:32:27.573301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.573183 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-52d131\"/\"kube-root-ca.crt\"" Oct 30 02:32:27.573301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.573210 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-52d131\"/\"openshift-service-ca.crt\"" Oct 30 02:32:27.623115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.621596 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:27.739114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.736963 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-model-cache\") pod \"test-model-52d131-kserve-644fd5978c-c6stv\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:27.739114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.737030 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76604767-dec2-42cd-b953-65221160999f-tls-certs\") pod \"test-model-52d131-kserve-644fd5978c-c6stv\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:27.739114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.737072 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-dshm\") pod \"test-model-52d131-kserve-644fd5978c-c6stv\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:27.739114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.737145 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zt8g\" (UniqueName: \"kubernetes.io/projected/76604767-dec2-42cd-b953-65221160999f-kube-api-access-2zt8g\") pod \"test-model-52d131-kserve-644fd5978c-c6stv\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:27.739114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.737202 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-home\") pod \"test-model-52d131-kserve-644fd5978c-c6stv\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:27.739114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.737273 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-tmp-dir\") pod \"test-model-52d131-kserve-644fd5978c-c6stv\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:27.841168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.838883 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-dshm\") pod \"test-model-52d131-kserve-644fd5978c-c6stv\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:27.841168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.838963 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zt8g\" (UniqueName: \"kubernetes.io/projected/76604767-dec2-42cd-b953-65221160999f-kube-api-access-2zt8g\") pod \"test-model-52d131-kserve-644fd5978c-c6stv\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:27.841168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.839033 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-home\") pod \"test-model-52d131-kserve-644fd5978c-c6stv\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:27.841168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.839116 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-tmp-dir\") pod \"test-model-52d131-kserve-644fd5978c-c6stv\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:27.841168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.839345 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-model-cache\") pod \"test-model-52d131-kserve-644fd5978c-c6stv\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:27.841168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.839388 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76604767-dec2-42cd-b953-65221160999f-tls-certs\") pod \"test-model-52d131-kserve-644fd5978c-c6stv\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:27.841168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.840401 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-home\") pod \"test-model-52d131-kserve-644fd5978c-c6stv\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:27.841168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.840989 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-tmp-dir\") pod \"test-model-52d131-kserve-644fd5978c-c6stv\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:27.842119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.841259 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-model-cache\") pod \"test-model-52d131-kserve-644fd5978c-c6stv\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:27.855499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.852546 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-dshm\") pod \"test-model-52d131-kserve-644fd5978c-c6stv\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:27.881110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.868503 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76604767-dec2-42cd-b953-65221160999f-tls-certs\") pod \"test-model-52d131-kserve-644fd5978c-c6stv\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:27.891111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:27.886387 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zt8g\" (UniqueName: \"kubernetes.io/projected/76604767-dec2-42cd-b953-65221160999f-kube-api-access-2zt8g\") pod \"test-model-52d131-kserve-644fd5978c-c6stv\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:28.171258 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:28.171029 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:28.582267 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:28.582182 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv"] Oct 30 02:32:28.585009 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:32:28.584981 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76604767_dec2_42cd_b953_65221160999f.slice/crio-36bdd021c4ae7ece82de99c4981ab0a03dcc4c248a0cd3563544f72a932df07c WatchSource:0}: Error finding container 36bdd021c4ae7ece82de99c4981ab0a03dcc4c248a0cd3563544f72a932df07c: Status 404 returned error can't find the container with id 36bdd021c4ae7ece82de99c4981ab0a03dcc4c248a0cd3563544f72a932df07c Oct 30 02:32:29.026657 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:29.021551 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:29.052644 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:29.051192 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9"] Oct 30 02:32:29.054472 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:29.054438 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d91bc928-pxfr9"] Oct 30 02:32:29.473343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:29.473305 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" event={"ID":"76604767-dec2-42cd-b953-65221160999f","Type":"ContainerStarted","Data":"d120ee65e7cb7507ccde5a41227f595de7ea1305580edc4c52527fd1b6218e7f"} Oct 30 02:32:29.473343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:29.473349 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" event={"ID":"76604767-dec2-42cd-b953-65221160999f","Type":"ContainerStarted","Data":"36bdd021c4ae7ece82de99c4981ab0a03dcc4c248a0cd3563544f72a932df07c"} Oct 30 02:32:29.515689 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:29.515393 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" podStartSLOduration=2.515376505 podStartE2EDuration="2.515376505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:32:29.514753144 +0000 UTC m=+3779.093544226" watchObservedRunningTime="2026-10-30 02:32:29.515376505 +0000 UTC m=+3779.094167585" Oct 30 02:32:30.172013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:30.171974 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:30.545791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:30.545704 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="429cee8e-bd15-4c14-800d-8e48377c9ec2" path="/var/lib/kubelet/pods/429cee8e-bd15-4c14-800d-8e48377c9ec2/volumes" Oct 30 02:32:30.615781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:30.615747 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:32.137188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:32.137144 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:33.512652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:33.512550 5998 generic.go:356] "Generic (PLEG): container finished" podID="1dac6d55-be03-4ce9-b8a7-a6397747e73d" containerID="3d9086dab4379432ea308ad7b591c989fb1b3b1da907058abd6c69aefd966c58" exitCode=0 Oct 30 02:32:33.513270 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:33.512632 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" event={"ID":"1dac6d55-be03-4ce9-b8a7-a6397747e73d","Type":"ContainerDied","Data":"3d9086dab4379432ea308ad7b591c989fb1b3b1da907058abd6c69aefd966c58"} Oct 30 02:32:33.513350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:33.513320 5998 scope.go:122] "RemoveContainer" containerID="3d9086dab4379432ea308ad7b591c989fb1b3b1da907058abd6c69aefd966c58" Oct 30 02:32:33.515716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:33.515679 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:32:33.515851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:33.515773 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:32:33.515851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:33.515823 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:32:33.715686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:33.715651 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:34.549008 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:34.548210 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" event={"ID":"1dac6d55-be03-4ce9-b8a7-a6397747e73d","Type":"ContainerStarted","Data":"2c195c6e29b1fbba708584386692afa2293b1bfb8d88c01105bd0f73e498be9d"} Oct 30 02:32:35.246163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:35.239792 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:32:35.246163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:35.239923 5998 patch_prober.go:29] interesting pod/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 02:32:35.246163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:35.239970 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" podUID="1dac6d55-be03-4ce9-b8a7-a6397747e73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 02:32:35.458886 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:35.458844 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:36.246272 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:36.245204 5998 patch_prober.go:29] interesting pod/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 02:32:36.246272 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:36.245262 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" podUID="1dac6d55-be03-4ce9-b8a7-a6397747e73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 02:32:36.716115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:36.712663 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:36.832115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:36.831185 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:37.113924 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.113830 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:37.242204 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.240554 5998 patch_prober.go:29] interesting pod/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 02:32:37.242204 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.240641 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" podUID="1dac6d55-be03-4ce9-b8a7-a6397747e73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 02:32:37.551314 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.551277 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb"] Oct 30 02:32:37.551927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.551907 5998 cpu_manager.go:414] "containerMap: removing container" podUID="046ebf9b-3d7e-4c67-bfcf-8545980b59ff" containerName="cleanup" Oct 30 02:32:37.551978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.551927 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="046ebf9b-3d7e-4c67-bfcf-8545980b59ff" containerName="cleanup" Oct 30 02:32:37.551978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.551970 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8e81c096-2704-4d9e-80b8-9b1f05e7151f" containerName="cleanup" Oct 30 02:32:37.551978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.551976 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e81c096-2704-4d9e-80b8-9b1f05e7151f" containerName="cleanup" Oct 30 02:32:37.552082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.551984 5998 cpu_manager.go:414] "containerMap: removing container" podUID="9e4909d0-f3ae-423a-a478-4ee5ce9734a1" containerName="cleanup" Oct 30 02:32:37.552082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.551989 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e4909d0-f3ae-423a-a478-4ee5ce9734a1" containerName="cleanup" Oct 30 02:32:37.552082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552000 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee" containerName="cleanup" Oct 30 02:32:37.552082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552005 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee" containerName="cleanup" Oct 30 02:32:37.552082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552012 5998 cpu_manager.go:414] "containerMap: removing container" podUID="e5882ea9-fe2e-4587-abb7-bb14939f53f6" containerName="cleanup" Oct 30 02:32:37.552082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552017 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5882ea9-fe2e-4587-abb7-bb14939f53f6" containerName="cleanup" Oct 30 02:32:37.552082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552026 5998 cpu_manager.go:414] "containerMap: removing container" podUID="2f3ead60-88ca-42af-9e25-7822589058e4" containerName="cleanup" Oct 30 02:32:37.552082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552030 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f3ead60-88ca-42af-9e25-7822589058e4" containerName="cleanup" Oct 30 02:32:37.552082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552036 5998 cpu_manager.go:414] "containerMap: removing container" podUID="96bc88e1-7b35-4b54-b12f-69227ae41a88" containerName="cleanup" Oct 30 02:32:37.552082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552040 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96bc88e1-7b35-4b54-b12f-69227ae41a88" containerName="cleanup" Oct 30 02:32:37.552082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552050 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c9b0ea41-0e61-456d-bbce-6ee7145552ea" containerName="cleanup" Oct 30 02:32:37.552082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552055 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b0ea41-0e61-456d-bbce-6ee7145552ea" containerName="cleanup" Oct 30 02:32:37.552082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552065 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7ff76844-137e-44ec-b146-4d9c84997ada" containerName="cleanup" Oct 30 02:32:37.552082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552070 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff76844-137e-44ec-b146-4d9c84997ada" containerName="cleanup" Oct 30 02:32:37.552082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552076 5998 cpu_manager.go:414] "containerMap: removing container" podUID="512be8ad-51e7-4919-a456-9fcfafab46bd" containerName="cleanup" Oct 30 02:32:37.552082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552081 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="512be8ad-51e7-4919-a456-9fcfafab46bd" containerName="cleanup" Oct 30 02:32:37.552628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552123 5998 cpu_manager.go:414] "containerMap: removing container" podUID="0771e950-74b9-45cb-a7be-ccb30c8c8c65" containerName="cleanup" Oct 30 02:32:37.552628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552130 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0771e950-74b9-45cb-a7be-ccb30c8c8c65" containerName="cleanup" Oct 30 02:32:37.552628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552138 5998 cpu_manager.go:414] "containerMap: removing container" podUID="be309ab3-73d7-42c0-92a8-0fc686504795" containerName="cleanup" Oct 30 02:32:37.552628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552143 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be309ab3-73d7-42c0-92a8-0fc686504795" containerName="cleanup" Oct 30 02:32:37.552628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552242 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="046ebf9b-3d7e-4c67-bfcf-8545980b59ff" containerName="cleanup" Oct 30 02:32:37.552628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552252 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f69dd9b-8eda-4368-b693-33c5b9721710" containerName="cleanup" Oct 30 02:32:37.552628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552261 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1411791d-2b8f-46e0-8b26-8799563d8c4b" containerName="istio-proxy" Oct 30 02:32:37.552628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552268 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e81c096-2704-4d9e-80b8-9b1f05e7151f" containerName="cleanup" Oct 30 02:32:37.552628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552275 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f4bafe4-34d9-4fb9-ac61-81d26a0ad4ee" containerName="cleanup" Oct 30 02:32:37.552628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552281 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5882ea9-fe2e-4587-abb7-bb14939f53f6" containerName="cleanup" Oct 30 02:32:37.552628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552288 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f3ead60-88ca-42af-9e25-7822589058e4" containerName="cleanup" Oct 30 02:32:37.552628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552295 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff76844-137e-44ec-b146-4d9c84997ada" containerName="cleanup" Oct 30 02:32:37.552628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552300 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="512be8ad-51e7-4919-a456-9fcfafab46bd" containerName="cleanup" Oct 30 02:32:37.552628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.552309 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="daaa2b88-a49a-4a88-8325-33307e969074" containerName="cleanup" Oct 30 02:32:37.555763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.555743 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb" Oct 30 02:32:37.571168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.571140 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:32:37.571287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.571200 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:32:37.581310 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.581276 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:32:37.583254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.583230 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:32:37.588080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.588049 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb"] Oct 30 02:32:37.655160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.655106 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv4wc\" (UniqueName: \"kubernetes.io/projected/fc1d4dfd-112c-410d-8f45-155b993865d7-kube-api-access-lv4wc\") pod \"maas-api-revoke-sub-cda67a7d-4m6hb\" (UID: \"fc1d4dfd-112c-410d-8f45-155b993865d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb" Oct 30 02:32:37.655380 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.655277 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc1d4dfd-112c-410d-8f45-155b993865d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cda67a7d-4m6hb\" (UID: \"fc1d4dfd-112c-410d-8f45-155b993865d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb" Oct 30 02:32:37.655504 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.655479 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc1d4dfd-112c-410d-8f45-155b993865d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cda67a7d-4m6hb\" (UID: \"fc1d4dfd-112c-410d-8f45-155b993865d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb" Oct 30 02:32:37.757571 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.757533 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc1d4dfd-112c-410d-8f45-155b993865d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cda67a7d-4m6hb\" (UID: \"fc1d4dfd-112c-410d-8f45-155b993865d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb" Oct 30 02:32:37.757791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.757691 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc1d4dfd-112c-410d-8f45-155b993865d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cda67a7d-4m6hb\" (UID: \"fc1d4dfd-112c-410d-8f45-155b993865d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb" Oct 30 02:32:37.757902 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.757885 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv4wc\" (UniqueName: \"kubernetes.io/projected/fc1d4dfd-112c-410d-8f45-155b993865d7-kube-api-access-lv4wc\") pod \"maas-api-revoke-sub-cda67a7d-4m6hb\" (UID: \"fc1d4dfd-112c-410d-8f45-155b993865d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb" Oct 30 02:32:37.758278 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.758233 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc1d4dfd-112c-410d-8f45-155b993865d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cda67a7d-4m6hb\" (UID: \"fc1d4dfd-112c-410d-8f45-155b993865d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb" Oct 30 02:32:37.760101 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.760063 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc1d4dfd-112c-410d-8f45-155b993865d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cda67a7d-4m6hb\" (UID: \"fc1d4dfd-112c-410d-8f45-155b993865d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb" Oct 30 02:32:37.781574 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.781544 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv4wc\" (UniqueName: \"kubernetes.io/projected/fc1d4dfd-112c-410d-8f45-155b993865d7-kube-api-access-lv4wc\") pod \"maas-api-revoke-sub-cda67a7d-4m6hb\" (UID: \"fc1d4dfd-112c-410d-8f45-155b993865d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb" Oct 30 02:32:37.870669 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:37.870570 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb" Oct 30 02:32:38.172449 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:38.172402 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:38.185797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:38.185767 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:38.248795 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:38.240454 5998 patch_prober.go:29] interesting pod/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 02:32:38.248795 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:38.240512 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" podUID="1dac6d55-be03-4ce9-b8a7-a6397747e73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 02:32:38.248795 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:38.244163 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb"] Oct 30 02:32:38.593449 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:38.593402 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb" event={"ID":"fc1d4dfd-112c-410d-8f45-155b993865d7","Type":"ContainerStarted","Data":"685ec4e2f5913be40787460e952ceb17fd6dcbf459b97852b2eb9a3f50fe1799"} Oct 30 02:32:38.639206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:38.639167 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:32:38.860052 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:38.859969 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:39.239695 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:39.239635 5998 patch_prober.go:29] interesting pod/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body= Oct 30 02:32:39.239930 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:39.239735 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" podUID="1dac6d55-be03-4ce9-b8a7-a6397747e73d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" Oct 30 02:32:39.492399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:39.492273 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:32:39.634114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:39.616750 5998 generic.go:356] "Generic (PLEG): container finished" podID="fc1d4dfd-112c-410d-8f45-155b993865d7" containerID="62e9b8507f69e73c7e522f4d6fa9262daa6d2ff4ae6d598219f50c54090f8fb5" exitCode=0 Oct 30 02:32:39.634114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:39.617593 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb" event={"ID":"fc1d4dfd-112c-410d-8f45-155b993865d7","Type":"ContainerDied","Data":"62e9b8507f69e73c7e522f4d6fa9262daa6d2ff4ae6d598219f50c54090f8fb5"} Oct 30 02:32:39.819058 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:39.818958 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:40.246467 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:40.246427 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:32:40.246884 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:40.246854 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:32:40.247816 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:40.247791 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:32:40.443759 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:40.443729 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:40.812901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:40.812867 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb" Oct 30 02:32:40.926527 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:40.926490 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1d4dfd-112c-410d-8f45-155b993865d7-kube-api-access-lv4wc\" (UniqueName: \"kubernetes.io/projected/fc1d4dfd-112c-410d-8f45-155b993865d7-kube-api-access-lv4wc\") pod \"fc1d4dfd-112c-410d-8f45-155b993865d7\" (UID: \"fc1d4dfd-112c-410d-8f45-155b993865d7\") " Oct 30 02:32:40.926710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:40.926564 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc1d4dfd-112c-410d-8f45-155b993865d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc1d4dfd-112c-410d-8f45-155b993865d7-maas-api-xks-ca\") pod \"fc1d4dfd-112c-410d-8f45-155b993865d7\" (UID: \"fc1d4dfd-112c-410d-8f45-155b993865d7\") " Oct 30 02:32:40.926804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:40.926785 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1d4dfd-112c-410d-8f45-155b993865d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc1d4dfd-112c-410d-8f45-155b993865d7-maas-api-openshift-ca\") pod \"fc1d4dfd-112c-410d-8f45-155b993865d7\" (UID: \"fc1d4dfd-112c-410d-8f45-155b993865d7\") " Oct 30 02:32:40.927148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:40.927119 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1d4dfd-112c-410d-8f45-155b993865d7-maas-api-openshift-ca" pod "fc1d4dfd-112c-410d-8f45-155b993865d7" (UID: "fc1d4dfd-112c-410d-8f45-155b993865d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:32:40.927403 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:40.927386 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc1d4dfd-112c-410d-8f45-155b993865d7-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:32:40.928676 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:40.928650 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc1d4dfd-112c-410d-8f45-155b993865d7-maas-api-xks-ca" pod "fc1d4dfd-112c-410d-8f45-155b993865d7" (UID: "fc1d4dfd-112c-410d-8f45-155b993865d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:32:40.928774 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:40.928675 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1d4dfd-112c-410d-8f45-155b993865d7-kube-api-access-lv4wc" pod "fc1d4dfd-112c-410d-8f45-155b993865d7" (UID: "fc1d4dfd-112c-410d-8f45-155b993865d7"). InnerVolumeSpecName "kube-api-access-lv4wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:32:41.029372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:41.029330 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv4wc\" (UniqueName: \"kubernetes.io/projected/fc1d4dfd-112c-410d-8f45-155b993865d7-kube-api-access-lv4wc\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:32:41.029372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:41.029366 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc1d4dfd-112c-410d-8f45-155b993865d7-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:32:41.636157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:41.636122 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb" Oct 30 02:32:41.636350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:41.636119 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb" event={"ID":"fc1d4dfd-112c-410d-8f45-155b993865d7","Type":"ContainerDied","Data":"685ec4e2f5913be40787460e952ceb17fd6dcbf459b97852b2eb9a3f50fe1799"} Oct 30 02:32:41.636350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:41.636252 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="685ec4e2f5913be40787460e952ceb17fd6dcbf459b97852b2eb9a3f50fe1799" Oct 30 02:32:42.095568 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:42.095534 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:43.034265 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:43.034193 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn"] Oct 30 02:32:43.051678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:43.051638 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-615952d9-vlngn"] Oct 30 02:32:43.800340 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:43.800268 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:44.539864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:44.539822 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae632736-734a-47bd-9987-dc725dbf3b11" path="/var/lib/kubelet/pods/ae632736-734a-47bd-9987-dc725dbf3b11/volumes" Oct 30 02:32:45.363173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:45.363141 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:47.115814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:47.115779 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:48.944865 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:48.944818 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:50.599416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:50.599381 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:52.291654 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:52.291624 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:53.033704 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:53.033661 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6"] Oct 30 02:32:53.043475 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:53.043441 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f00a4123-txnq6"] Oct 30 02:32:53.844418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:53.844378 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:54.543900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:54.543858 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1df1953-76a5-4cb5-a2d1-0fc791c6a27c" path="/var/lib/kubelet/pods/a1df1953-76a5-4cb5-a2d1-0fc791c6a27c/volumes" Oct 30 02:32:55.121157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.121116 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7"] Oct 30 02:32:55.122190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.122150 5998 cpu_manager.go:414] "containerMap: removing container" podUID="fc1d4dfd-112c-410d-8f45-155b993865d7" containerName="revoke-keys" Oct 30 02:32:55.122190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.122178 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1d4dfd-112c-410d-8f45-155b993865d7" containerName="revoke-keys" Oct 30 02:32:55.122367 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.122355 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1d4dfd-112c-410d-8f45-155b993865d7" containerName="revoke-keys" Oct 30 02:32:55.127981 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.127956 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.147385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.147348 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7"] Oct 30 02:32:55.152445 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.152414 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-07fb25\"/\"openshift-service-ca.crt\"" Oct 30 02:32:55.153115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.152717 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-07fb25\"/\"default-dockercfg-6tzzx\"" Oct 30 02:32:55.153293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.152797 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-07fb25\"/\"test-model-07fb25-kserve-self-signed-certs\"" Oct 30 02:32:55.153417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.153035 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-07fb25\"/\"kube-root-ca.crt\"" Oct 30 02:32:55.186042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.186003 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-tmp-dir\") pod \"test-model-07fb25-kserve-646d9d44cf-qx8p7\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.186476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.186446 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-model-cache\") pod \"test-model-07fb25-kserve-646d9d44cf-qx8p7\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.186641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.186618 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kkgx\" (UniqueName: \"kubernetes.io/projected/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-kube-api-access-2kkgx\") pod \"test-model-07fb25-kserve-646d9d44cf-qx8p7\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.186879 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.186859 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-home\") pod \"test-model-07fb25-kserve-646d9d44cf-qx8p7\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.187005 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.186988 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-dshm\") pod \"test-model-07fb25-kserve-646d9d44cf-qx8p7\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.187255 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.187233 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-tls-certs\") pod \"test-model-07fb25-kserve-646d9d44cf-qx8p7\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.188702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.188673 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:55.251016 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:32:55.250971 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:32:55.288873 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.288838 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-model-cache\") pod \"test-model-07fb25-kserve-646d9d44cf-qx8p7\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.289081 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.288890 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kkgx\" (UniqueName: \"kubernetes.io/projected/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-kube-api-access-2kkgx\") pod \"test-model-07fb25-kserve-646d9d44cf-qx8p7\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.289081 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.289031 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-home\") pod \"test-model-07fb25-kserve-646d9d44cf-qx8p7\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.289081 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.289054 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-dshm\") pod \"test-model-07fb25-kserve-646d9d44cf-qx8p7\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.289348 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.289181 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-tls-certs\") pod \"test-model-07fb25-kserve-646d9d44cf-qx8p7\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.289348 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.289263 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-tmp-dir\") pod \"test-model-07fb25-kserve-646d9d44cf-qx8p7\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.289348 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.289343 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-model-cache\") pod \"test-model-07fb25-kserve-646d9d44cf-qx8p7\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.289526 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.289467 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-home\") pod \"test-model-07fb25-kserve-646d9d44cf-qx8p7\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.289750 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.289722 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-tmp-dir\") pod \"test-model-07fb25-kserve-646d9d44cf-qx8p7\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.291879 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.291853 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-tls-certs\") pod \"test-model-07fb25-kserve-646d9d44cf-qx8p7\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.291986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.291928 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-dshm\") pod \"test-model-07fb25-kserve-646d9d44cf-qx8p7\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.313490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.313450 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kkgx\" (UniqueName: \"kubernetes.io/projected/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-kube-api-access-2kkgx\") pod \"test-model-07fb25-kserve-646d9d44cf-qx8p7\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.458191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.457545 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:55.541365 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.534413 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:55.914419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:55.914383 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7"] Oct 30 02:32:55.916569 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:32:55.916541 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ced29b5_0313_45a9_8a67_54e28ebc4bf2.slice/crio-5f6d16875a8f07be1bfce3ebcd8c0c7029de96e92fbc5e62602acc2cedc5abff WatchSource:0}: Error finding container 5f6d16875a8f07be1bfce3ebcd8c0c7029de96e92fbc5e62602acc2cedc5abff: Status 404 returned error can't find the container with id 5f6d16875a8f07be1bfce3ebcd8c0c7029de96e92fbc5e62602acc2cedc5abff Oct 30 02:32:56.767752 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:56.767715 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" event={"ID":"9ced29b5-0313-45a9-8a67-54e28ebc4bf2","Type":"ContainerStarted","Data":"5329e1e78cd2ea5551454685834db8b69319b552d21094794da3798403fee788"} Oct 30 02:32:56.767752 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:56.767758 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" event={"ID":"9ced29b5-0313-45a9-8a67-54e28ebc4bf2","Type":"ContainerStarted","Data":"5f6d16875a8f07be1bfce3ebcd8c0c7029de96e92fbc5e62602acc2cedc5abff"} Oct 30 02:32:56.810555 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:56.810498 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" podStartSLOduration=1.810480232 podStartE2EDuration="1.810480232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:32:56.806142417 +0000 UTC m=+3806.384933499" watchObservedRunningTime="2026-10-30 02:32:56.810480232 +0000 UTC m=+3806.389271313" Oct 30 02:32:57.299894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:57.299849 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj"] Oct 30 02:32:57.327050 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:57.327009 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:57.457117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:57.456650 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:57.473114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:57.466124 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:32:57.861576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:57.861542 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz"] Oct 30 02:32:57.865963 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:57.865942 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz" Oct 30 02:32:57.878992 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:57.878960 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:32:57.879203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:57.879184 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:32:57.880247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:57.880227 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:32:57.889371 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:57.889350 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:32:57.895527 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:57.895490 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz"] Oct 30 02:32:58.035711 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:58.035656 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a8b10b9-ttrtz\" (UID: \"7e33e5cb-5be4-4ac8-ab50-506cac62ae3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz" Oct 30 02:32:58.036070 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:58.036044 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a8b10b9-ttrtz\" (UID: \"7e33e5cb-5be4-4ac8-ab50-506cac62ae3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz" Oct 30 02:32:58.036212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:58.036187 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p4hz\" (UniqueName: \"kubernetes.io/projected/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-kube-api-access-9p4hz\") pod \"maas-api-revoke-sub-0a8b10b9-ttrtz\" (UID: \"7e33e5cb-5be4-4ac8-ab50-506cac62ae3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz" Oct 30 02:32:58.137807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:58.137695 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a8b10b9-ttrtz\" (UID: \"7e33e5cb-5be4-4ac8-ab50-506cac62ae3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz" Oct 30 02:32:58.137807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:58.137776 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p4hz\" (UniqueName: \"kubernetes.io/projected/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-kube-api-access-9p4hz\") pod \"maas-api-revoke-sub-0a8b10b9-ttrtz\" (UID: \"7e33e5cb-5be4-4ac8-ab50-506cac62ae3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz" Oct 30 02:32:58.138081 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:58.137847 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a8b10b9-ttrtz\" (UID: \"7e33e5cb-5be4-4ac8-ab50-506cac62ae3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz" Oct 30 02:32:58.138560 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:58.138533 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a8b10b9-ttrtz\" (UID: \"7e33e5cb-5be4-4ac8-ab50-506cac62ae3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz" Oct 30 02:32:58.140438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:58.140403 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a8b10b9-ttrtz\" (UID: \"7e33e5cb-5be4-4ac8-ab50-506cac62ae3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz" Oct 30 02:32:58.159749 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:58.159716 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p4hz\" (UniqueName: \"kubernetes.io/projected/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-kube-api-access-9p4hz\") pod \"maas-api-revoke-sub-0a8b10b9-ttrtz\" (UID: \"7e33e5cb-5be4-4ac8-ab50-506cac62ae3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz" Oct 30 02:32:58.181697 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:58.181666 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz" Oct 30 02:32:58.504143 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:58.504106 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz"] Oct 30 02:32:58.508407 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:32:58.508373 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e33e5cb_5be4_4ac8_ab50_506cac62ae3f.slice/crio-0094c41598eeb41c5009b0d88d63c2a390af9005bac2e74f82e311d76be30926 WatchSource:0}: Error finding container 0094c41598eeb41c5009b0d88d63c2a390af9005bac2e74f82e311d76be30926: Status 404 returned error can't find the container with id 0094c41598eeb41c5009b0d88d63c2a390af9005bac2e74f82e311d76be30926 Oct 30 02:32:58.788640 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:58.788538 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz" event={"ID":"7e33e5cb-5be4-4ac8-ab50-506cac62ae3f","Type":"ContainerStarted","Data":"0094c41598eeb41c5009b0d88d63c2a390af9005bac2e74f82e311d76be30926"} Oct 30 02:32:58.984806 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:58.984766 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:32:59.798627 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:59.798590 5998 generic.go:356] "Generic (PLEG): container finished" podID="7e33e5cb-5be4-4ac8-ab50-506cac62ae3f" containerID="cc8c26f63c0f966f8f71bc241fe275f34e98278148c264a8643f6a570a9bb157" exitCode=0 Oct 30 02:32:59.798825 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:32:59.798669 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz" event={"ID":"7e33e5cb-5be4-4ac8-ab50-506cac62ae3f","Type":"ContainerDied","Data":"cc8c26f63c0f966f8f71bc241fe275f34e98278148c264a8643f6a570a9bb157"} Oct 30 02:33:00.495990 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:00.495958 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:00.973526 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:00.973498 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz" Oct 30 02:33:00.986147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:00.986106 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-kube-api-access-9p4hz\" (UniqueName: \"kubernetes.io/projected/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-kube-api-access-9p4hz\") pod \"7e33e5cb-5be4-4ac8-ab50-506cac62ae3f\" (UID: \"7e33e5cb-5be4-4ac8-ab50-506cac62ae3f\") " Oct 30 02:33:00.986320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:00.986262 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-maas-api-xks-ca\") pod \"7e33e5cb-5be4-4ac8-ab50-506cac62ae3f\" (UID: \"7e33e5cb-5be4-4ac8-ab50-506cac62ae3f\") " Oct 30 02:33:00.986474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:00.986458 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-maas-api-openshift-ca\") pod \"7e33e5cb-5be4-4ac8-ab50-506cac62ae3f\" (UID: \"7e33e5cb-5be4-4ac8-ab50-506cac62ae3f\") " Oct 30 02:33:00.986877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:00.986845 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-maas-api-openshift-ca" pod "7e33e5cb-5be4-4ac8-ab50-506cac62ae3f" (UID: "7e33e5cb-5be4-4ac8-ab50-506cac62ae3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:33:00.987288 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:00.987263 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:00.988388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:00.988367 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-maas-api-xks-ca" pod "7e33e5cb-5be4-4ac8-ab50-506cac62ae3f" (UID: "7e33e5cb-5be4-4ac8-ab50-506cac62ae3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:33:00.988468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:00.988425 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-kube-api-access-9p4hz" pod "7e33e5cb-5be4-4ac8-ab50-506cac62ae3f" (UID: "7e33e5cb-5be4-4ac8-ab50-506cac62ae3f"). InnerVolumeSpecName "kube-api-access-9p4hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:33:01.088912 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:01.088825 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:01.088912 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:01.088856 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p4hz\" (UniqueName: \"kubernetes.io/projected/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f-kube-api-access-9p4hz\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:01.816626 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:01.816589 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz" Oct 30 02:33:01.816626 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:01.816611 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz" event={"ID":"7e33e5cb-5be4-4ac8-ab50-506cac62ae3f","Type":"ContainerDied","Data":"0094c41598eeb41c5009b0d88d63c2a390af9005bac2e74f82e311d76be30926"} Oct 30 02:33:01.817114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:01.816665 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0094c41598eeb41c5009b0d88d63c2a390af9005bac2e74f82e311d76be30926" Oct 30 02:33:02.202391 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:02.201674 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:03.877125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:03.877079 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:05.393444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:05.393407 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:05.473100 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:05.473048 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:33:05.496079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:05.496048 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:33:05.873949 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:05.873919 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:33:07.092494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:07.092444 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:08.627148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:08.627114 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:10.219667 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:10.219633 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:11.875691 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:11.875627 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:12.354222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.354177 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" podUID="44d99268-92b1-4b8b-8061-4b5732b2de40" containerName="main" containerID="cri-o://31e66cf31f29371c6a5628d93b23258af9effca76a8a93437648269a969a6982" gracePeriod=45 Oct 30 02:33:12.682362 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.682322 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:33:12.798552 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.798514 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44d99268-92b1-4b8b-8061-4b5732b2de40-tls-certs\" (UniqueName: \"kubernetes.io/secret/44d99268-92b1-4b8b-8061-4b5732b2de40-tls-certs\") pod \"44d99268-92b1-4b8b-8061-4b5732b2de40\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " Oct 30 02:33:12.798742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.798663 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d99268-92b1-4b8b-8061-4b5732b2de40-kube-api-access-bfd89\" (UniqueName: \"kubernetes.io/projected/44d99268-92b1-4b8b-8061-4b5732b2de40-kube-api-access-bfd89\") pod \"44d99268-92b1-4b8b-8061-4b5732b2de40\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " Oct 30 02:33:12.798742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.798740 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-home\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-home\") pod \"44d99268-92b1-4b8b-8061-4b5732b2de40\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " Oct 30 02:33:12.798882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.798810 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-dshm\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-dshm\") pod \"44d99268-92b1-4b8b-8061-4b5732b2de40\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " Oct 30 02:33:12.798882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.798851 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-model-cache\") pod \"44d99268-92b1-4b8b-8061-4b5732b2de40\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " Oct 30 02:33:12.799040 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.799020 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-tmp-dir\") pod \"44d99268-92b1-4b8b-8061-4b5732b2de40\" (UID: \"44d99268-92b1-4b8b-8061-4b5732b2de40\") " Oct 30 02:33:12.799289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.799026 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-home" pod "44d99268-92b1-4b8b-8061-4b5732b2de40" (UID: "44d99268-92b1-4b8b-8061-4b5732b2de40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:33:12.799422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.799218 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-model-cache" pod "44d99268-92b1-4b8b-8061-4b5732b2de40" (UID: "44d99268-92b1-4b8b-8061-4b5732b2de40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:33:12.799422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.799230 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-tmp-dir" pod "44d99268-92b1-4b8b-8061-4b5732b2de40" (UID: "44d99268-92b1-4b8b-8061-4b5732b2de40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:33:12.800062 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.800033 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:12.800144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.800068 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:12.800144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.800083 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:12.801408 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.801383 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-dshm" pod "44d99268-92b1-4b8b-8061-4b5732b2de40" (UID: "44d99268-92b1-4b8b-8061-4b5732b2de40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:33:12.801539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.801439 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d99268-92b1-4b8b-8061-4b5732b2de40-kube-api-access-bfd89" pod "44d99268-92b1-4b8b-8061-4b5732b2de40" (UID: "44d99268-92b1-4b8b-8061-4b5732b2de40"). InnerVolumeSpecName "kube-api-access-bfd89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:33:12.805557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.805528 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44d99268-92b1-4b8b-8061-4b5732b2de40-tls-certs" pod "44d99268-92b1-4b8b-8061-4b5732b2de40" (UID: "44d99268-92b1-4b8b-8061-4b5732b2de40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:33:12.901776 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.901693 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44d99268-92b1-4b8b-8061-4b5732b2de40-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:12.901776 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.901726 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfd89\" (UniqueName: \"kubernetes.io/projected/44d99268-92b1-4b8b-8061-4b5732b2de40-kube-api-access-bfd89\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:12.901776 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.901738 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44d99268-92b1-4b8b-8061-4b5732b2de40-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:12.939946 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.939895 5998 generic.go:356] "Generic (PLEG): container finished" podID="44d99268-92b1-4b8b-8061-4b5732b2de40" containerID="31e66cf31f29371c6a5628d93b23258af9effca76a8a93437648269a969a6982" exitCode=0 Oct 30 02:33:12.940220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.939947 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" Oct 30 02:33:12.940220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.939989 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" event={"ID":"44d99268-92b1-4b8b-8061-4b5732b2de40","Type":"ContainerDied","Data":"31e66cf31f29371c6a5628d93b23258af9effca76a8a93437648269a969a6982"} Oct 30 02:33:12.940220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.940063 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj" event={"ID":"44d99268-92b1-4b8b-8061-4b5732b2de40","Type":"ContainerDied","Data":"e73f947794c4e7b65dea1fea9e4f1b0678afb391e3345527c25d6a9726b21591"} Oct 30 02:33:12.940452 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.940427 5998 scope.go:122] "RemoveContainer" containerID="31e66cf31f29371c6a5628d93b23258af9effca76a8a93437648269a969a6982" Oct 30 02:33:12.953139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.953117 5998 scope.go:122] "RemoveContainer" containerID="31e66cf31f29371c6a5628d93b23258af9effca76a8a93437648269a969a6982" Oct 30 02:33:12.953443 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:33:12.953418 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e66cf31f29371c6a5628d93b23258af9effca76a8a93437648269a969a6982\": container with ID starting with 31e66cf31f29371c6a5628d93b23258af9effca76a8a93437648269a969a6982 not found: ID does not exist" containerID="31e66cf31f29371c6a5628d93b23258af9effca76a8a93437648269a969a6982" Oct 30 02:33:12.953536 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.953450 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e66cf31f29371c6a5628d93b23258af9effca76a8a93437648269a969a6982"} err="failed to get container status \"31e66cf31f29371c6a5628d93b23258af9effca76a8a93437648269a969a6982\": rpc error: code = NotFound desc = could not find container \"31e66cf31f29371c6a5628d93b23258af9effca76a8a93437648269a969a6982\": container with ID starting with 31e66cf31f29371c6a5628d93b23258af9effca76a8a93437648269a969a6982 not found: ID does not exist" Oct 30 02:33:12.997777 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:12.997743 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj"] Oct 30 02:33:13.001273 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:13.001245 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3d5da6/test-model-3d5da6-kserve-846bd97fbf-qdpwj"] Oct 30 02:33:13.460938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:13.460901 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:14.543663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:14.543626 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d99268-92b1-4b8b-8061-4b5732b2de40" path="/var/lib/kubelet/pods/44d99268-92b1-4b8b-8061-4b5732b2de40/volumes" Oct 30 02:33:15.094968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:15.094931 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:16.661083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:16.661042 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:18.470923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:18.470878 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:19.993967 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:19.993924 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:21.175307 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:21.175236 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x"] Oct 30 02:33:21.209804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:21.209765 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:21.602506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:21.602355 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:21.713810 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:21.713773 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:23.184245 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:23.184211 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:24.694061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:24.694025 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:25.219880 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:33:25.219831 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:33:26.171119 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:26.171055 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:26.281387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:26.281348 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:26.391689 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:26.391646 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:27.000354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:27.000314 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:27.120582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:27.120531 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:28.424824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:28.424779 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:28.996752 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:28.996713 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:30.081763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:30.081697 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:32.089716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:32.089609 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:33.915104 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:33.915053 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:34.211418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:34.211324 5998 scope.go:122] "RemoveContainer" containerID="dcab84bc209875a52fced0017bc2b0c70016cd4e31c62f3014759c4759525e1f" Oct 30 02:33:34.228524 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:34.228501 5998 scope.go:122] "RemoveContainer" containerID="8e9dea2da59e462a4892b85e49babd3c694f73154fcaf5fda80322ccafd8bcc4" Oct 30 02:33:35.533835 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:35.533794 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:35.653293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:35.653232 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:36.234604 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.234559 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" podUID="14b0f602-68d2-4494-a92f-c6df12bcdaad" containerName="main" containerID="cri-o://d24c607f64f2863560cb023e70d6a4e2bef540daad0b86a4fb6612121a66d11e" gracePeriod=45 Oct 30 02:33:36.556132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.556101 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:33:36.669756 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.669714 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-home\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-home\") pod \"14b0f602-68d2-4494-a92f-c6df12bcdaad\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " Oct 30 02:33:36.669976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.669798 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-model-cache\") pod \"14b0f602-68d2-4494-a92f-c6df12bcdaad\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " Oct 30 02:33:36.669976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.669850 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-tmp-dir\") pod \"14b0f602-68d2-4494-a92f-c6df12bcdaad\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " Oct 30 02:33:36.670077 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.670048 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b0f602-68d2-4494-a92f-c6df12bcdaad-kube-api-access-lcq4l\" (UniqueName: \"kubernetes.io/projected/14b0f602-68d2-4494-a92f-c6df12bcdaad-kube-api-access-lcq4l\") pod \"14b0f602-68d2-4494-a92f-c6df12bcdaad\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " Oct 30 02:33:36.670199 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.670078 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-home" pod "14b0f602-68d2-4494-a92f-c6df12bcdaad" (UID: "14b0f602-68d2-4494-a92f-c6df12bcdaad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:33:36.670199 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.670137 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-dshm\") pod \"14b0f602-68d2-4494-a92f-c6df12bcdaad\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " Oct 30 02:33:36.670199 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.670155 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-model-cache" pod "14b0f602-68d2-4494-a92f-c6df12bcdaad" (UID: "14b0f602-68d2-4494-a92f-c6df12bcdaad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:33:36.670380 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.670223 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14b0f602-68d2-4494-a92f-c6df12bcdaad-tls-certs\" (UniqueName: \"kubernetes.io/secret/14b0f602-68d2-4494-a92f-c6df12bcdaad-tls-certs\") pod \"14b0f602-68d2-4494-a92f-c6df12bcdaad\" (UID: \"14b0f602-68d2-4494-a92f-c6df12bcdaad\") " Oct 30 02:33:36.670380 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.670342 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-tmp-dir" pod "14b0f602-68d2-4494-a92f-c6df12bcdaad" (UID: "14b0f602-68d2-4494-a92f-c6df12bcdaad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:33:36.672019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.671490 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:36.672019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.671629 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:36.672019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.671656 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:36.672877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.672847 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b0f602-68d2-4494-a92f-c6df12bcdaad-tls-certs" pod "14b0f602-68d2-4494-a92f-c6df12bcdaad" (UID: "14b0f602-68d2-4494-a92f-c6df12bcdaad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:33:36.672877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.672854 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b0f602-68d2-4494-a92f-c6df12bcdaad-kube-api-access-lcq4l" pod "14b0f602-68d2-4494-a92f-c6df12bcdaad" (UID: "14b0f602-68d2-4494-a92f-c6df12bcdaad"). InnerVolumeSpecName "kube-api-access-lcq4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:33:36.673935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.673914 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-dshm" pod "14b0f602-68d2-4494-a92f-c6df12bcdaad" (UID: "14b0f602-68d2-4494-a92f-c6df12bcdaad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:33:36.773705 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.773621 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14b0f602-68d2-4494-a92f-c6df12bcdaad-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:36.773705 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.773654 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14b0f602-68d2-4494-a92f-c6df12bcdaad-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:36.773705 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.773665 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcq4l\" (UniqueName: \"kubernetes.io/projected/14b0f602-68d2-4494-a92f-c6df12bcdaad-kube-api-access-lcq4l\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:36.962248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:36.962200 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:37.355117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:37.354716 5998 generic.go:356] "Generic (PLEG): container finished" podID="14b0f602-68d2-4494-a92f-c6df12bcdaad" containerID="d24c607f64f2863560cb023e70d6a4e2bef540daad0b86a4fb6612121a66d11e" exitCode=0 Oct 30 02:33:37.355117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:37.355053 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" event={"ID":"14b0f602-68d2-4494-a92f-c6df12bcdaad","Type":"ContainerDied","Data":"d24c607f64f2863560cb023e70d6a4e2bef540daad0b86a4fb6612121a66d11e"} Oct 30 02:33:37.355117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:37.355115 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" event={"ID":"14b0f602-68d2-4494-a92f-c6df12bcdaad","Type":"ContainerDied","Data":"1fcd21bf4f11e21bb64f0391abdb5fd3038c986c8f3f2dca68bf51bd5ab87e29"} Oct 30 02:33:37.355464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:37.355141 5998 scope.go:122] "RemoveContainer" containerID="d24c607f64f2863560cb023e70d6a4e2bef540daad0b86a4fb6612121a66d11e" Oct 30 02:33:37.355464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:37.355326 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x" Oct 30 02:33:37.576016 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:37.574315 5998 scope.go:122] "RemoveContainer" containerID="d24c607f64f2863560cb023e70d6a4e2bef540daad0b86a4fb6612121a66d11e" Oct 30 02:33:37.576016 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:33:37.574745 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d24c607f64f2863560cb023e70d6a4e2bef540daad0b86a4fb6612121a66d11e\": container with ID starting with d24c607f64f2863560cb023e70d6a4e2bef540daad0b86a4fb6612121a66d11e not found: ID does not exist" containerID="d24c607f64f2863560cb023e70d6a4e2bef540daad0b86a4fb6612121a66d11e" Oct 30 02:33:37.576016 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:37.574784 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d24c607f64f2863560cb023e70d6a4e2bef540daad0b86a4fb6612121a66d11e"} err="failed to get container status \"d24c607f64f2863560cb023e70d6a4e2bef540daad0b86a4fb6612121a66d11e\": rpc error: code = NotFound desc = could not find container \"d24c607f64f2863560cb023e70d6a4e2bef540daad0b86a4fb6612121a66d11e\": container with ID starting with d24c607f64f2863560cb023e70d6a4e2bef540daad0b86a4fb6612121a66d11e not found: ID does not exist" Oct 30 02:33:37.600994 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:37.600161 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x"] Oct 30 02:33:37.607169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:37.603420 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d97e4c/test-model-d97e4c-kserve-66465b5c75-zgz5x"] Oct 30 02:33:37.758123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:37.753299 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:38.552933 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:38.552889 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14b0f602-68d2-4494-a92f-c6df12bcdaad" path="/var/lib/kubelet/pods/14b0f602-68d2-4494-a92f-c6df12bcdaad/volumes" Oct 30 02:33:39.516627 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:39.516589 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:39.934636 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:39.934599 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:41.065317 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:41.065221 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:41.740063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:41.740027 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:41.856831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:41.856792 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:42.575654 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:42.575618 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:42.901292 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:42.901207 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:43.005813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:43.005779 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5"] Oct 30 02:33:43.006027 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:43.005979 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" podUID="1d926a5c-800a-4b12-b134-be5c4ff7a619" containerName="istio-proxy" containerID="cri-o://42f311f1a64586e4611f77a013b44c896755cc928b6d060272f725f35dec219e" gracePeriod=30 Oct 30 02:33:43.042850 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:43.042816 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:43.743576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:43.743537 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:44.731471 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:44.731430 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:45.403696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:45.403654 5998 patch_prober.go:29] interesting pod/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:33:45.404149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:45.403732 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" podUID="1d926a5c-800a-4b12-b134-be5c4ff7a619" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:33:46.344003 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:46.343965 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:48.066187 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.066152 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:48.351891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.351857 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:33:48.467457 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.467413 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-data\") pod \"1d926a5c-800a-4b12-b134-be5c4ff7a619\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " Oct 30 02:33:48.467663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.467497 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-credential-socket\") pod \"1d926a5c-800a-4b12-b134-be5c4ff7a619\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " Oct 30 02:33:48.467663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.467594 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-podinfo\") pod \"1d926a5c-800a-4b12-b134-be5c4ff7a619\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " Oct 30 02:33:48.467782 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.467699 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-data" pod "1d926a5c-800a-4b12-b134-be5c4ff7a619" (UID: "1d926a5c-800a-4b12-b134-be5c4ff7a619"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:33:48.467782 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.467761 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-envoy\") pod \"1d926a5c-800a-4b12-b134-be5c4ff7a619\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " Oct 30 02:33:48.467895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.467791 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-credential-socket" pod "1d926a5c-800a-4b12-b134-be5c4ff7a619" (UID: "1d926a5c-800a-4b12-b134-be5c4ff7a619"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:33:48.467895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.467803 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d926a5c-800a-4b12-b134-be5c4ff7a619-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d926a5c-800a-4b12-b134-be5c4ff7a619-istiod-ca-cert\") pod \"1d926a5c-800a-4b12-b134-be5c4ff7a619\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " Oct 30 02:33:48.468012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.467924 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-token\" (UniqueName: \"kubernetes.io/projected/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-token\") pod \"1d926a5c-800a-4b12-b134-be5c4ff7a619\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " Oct 30 02:33:48.468012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.467961 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-workload-certs\") pod \"1d926a5c-800a-4b12-b134-be5c4ff7a619\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " Oct 30 02:33:48.468012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.467999 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d926a5c-800a-4b12-b134-be5c4ff7a619-kube-api-access-8vw87\" (UniqueName: \"kubernetes.io/projected/1d926a5c-800a-4b12-b134-be5c4ff7a619-kube-api-access-8vw87\") pod \"1d926a5c-800a-4b12-b134-be5c4ff7a619\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " Oct 30 02:33:48.468231 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.468114 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-workload-socket\") pod \"1d926a5c-800a-4b12-b134-be5c4ff7a619\" (UID: \"1d926a5c-800a-4b12-b134-be5c4ff7a619\") " Oct 30 02:33:48.468302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.468213 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d926a5c-800a-4b12-b134-be5c4ff7a619-istiod-ca-cert" pod "1d926a5c-800a-4b12-b134-be5c4ff7a619" (UID: "1d926a5c-800a-4b12-b134-be5c4ff7a619"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:33:48.468554 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.468529 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-workload-socket" pod "1d926a5c-800a-4b12-b134-be5c4ff7a619" (UID: "1d926a5c-800a-4b12-b134-be5c4ff7a619"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:33:48.468634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.468558 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-workload-certs" pod "1d926a5c-800a-4b12-b134-be5c4ff7a619" (UID: "1d926a5c-800a-4b12-b134-be5c4ff7a619"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:33:48.468864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.468841 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d926a5c-800a-4b12-b134-be5c4ff7a619-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:48.468864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.468864 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:48.469009 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.468873 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:48.469009 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.468882 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:48.469009 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.468890 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:48.470157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.470124 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-envoy" pod "1d926a5c-800a-4b12-b134-be5c4ff7a619" (UID: "1d926a5c-800a-4b12-b134-be5c4ff7a619"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:33:48.470429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.470401 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-podinfo" pod "1d926a5c-800a-4b12-b134-be5c4ff7a619" (UID: "1d926a5c-800a-4b12-b134-be5c4ff7a619"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:33:48.470522 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.470458 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-token" pod "1d926a5c-800a-4b12-b134-be5c4ff7a619" (UID: "1d926a5c-800a-4b12-b134-be5c4ff7a619"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:33:48.470573 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.470549 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d926a5c-800a-4b12-b134-be5c4ff7a619-kube-api-access-8vw87" pod "1d926a5c-800a-4b12-b134-be5c4ff7a619" (UID: "1d926a5c-800a-4b12-b134-be5c4ff7a619"). InnerVolumeSpecName "kube-api-access-8vw87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:33:48.491755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.491724 5998 generic.go:356] "Generic (PLEG): container finished" podID="1d926a5c-800a-4b12-b134-be5c4ff7a619" containerID="42f311f1a64586e4611f77a013b44c896755cc928b6d060272f725f35dec219e" exitCode=0 Oct 30 02:33:48.491895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.491765 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" Oct 30 02:33:48.491895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.491765 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" event={"ID":"1d926a5c-800a-4b12-b134-be5c4ff7a619","Type":"ContainerDied","Data":"42f311f1a64586e4611f77a013b44c896755cc928b6d060272f725f35dec219e"} Oct 30 02:33:48.491895 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.491856 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5" event={"ID":"1d926a5c-800a-4b12-b134-be5c4ff7a619","Type":"ContainerDied","Data":"800a41e1bc5c1a449ad452dd347231c7487b5ea1117bf2c14431053e9be5de72"} Oct 30 02:33:48.492081 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.491900 5998 scope.go:122] "RemoveContainer" containerID="42f311f1a64586e4611f77a013b44c896755cc928b6d060272f725f35dec219e" Oct 30 02:33:48.505265 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.505247 5998 scope.go:122] "RemoveContainer" containerID="42f311f1a64586e4611f77a013b44c896755cc928b6d060272f725f35dec219e" Oct 30 02:33:48.505548 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:33:48.505523 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42f311f1a64586e4611f77a013b44c896755cc928b6d060272f725f35dec219e\": container with ID starting with 42f311f1a64586e4611f77a013b44c896755cc928b6d060272f725f35dec219e not found: ID does not exist" containerID="42f311f1a64586e4611f77a013b44c896755cc928b6d060272f725f35dec219e" Oct 30 02:33:48.505628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.505556 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42f311f1a64586e4611f77a013b44c896755cc928b6d060272f725f35dec219e"} err="failed to get container status \"42f311f1a64586e4611f77a013b44c896755cc928b6d060272f725f35dec219e\": rpc error: code = NotFound desc = could not find container \"42f311f1a64586e4611f77a013b44c896755cc928b6d060272f725f35dec219e\": container with ID starting with 42f311f1a64586e4611f77a013b44c896755cc928b6d060272f725f35dec219e not found: ID does not exist" Oct 30 02:33:48.543262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.543218 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5"] Oct 30 02:33:48.546005 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.545973 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3d5da6-openshift-default-df85545b9-2b6l5"] Oct 30 02:33:48.570570 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.570499 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:48.570570 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.570524 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:48.570570 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.570534 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vw87\" (UniqueName: \"kubernetes.io/projected/1d926a5c-800a-4b12-b134-be5c4ff7a619-kube-api-access-8vw87\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:48.570570 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:48.570544 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d926a5c-800a-4b12-b134-be5c4ff7a619-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:33:49.677519 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:49.677448 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:49.885436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:49.885397 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:49.999978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:49.999895 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:50.623855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:50.623817 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d926a5c-800a-4b12-b134-be5c4ff7a619" path="/var/lib/kubelet/pods/1d926a5c-800a-4b12-b134-be5c4ff7a619/volumes" Oct 30 02:33:50.930079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:50.930038 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:51.397487 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:51.397398 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:52.908484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:52.908447 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:54.600359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:54.600319 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:55.219248 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:33:55.219211 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:33:56.276747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:56.276709 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:57.342694 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.342652 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx"] Oct 30 02:33:57.343666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.343645 5998 cpu_manager.go:414] "containerMap: removing container" podUID="14b0f602-68d2-4494-a92f-c6df12bcdaad" containerName="main" Oct 30 02:33:57.343730 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.343669 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b0f602-68d2-4494-a92f-c6df12bcdaad" containerName="main" Oct 30 02:33:57.343730 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.343693 5998 cpu_manager.go:414] "containerMap: removing container" podUID="44d99268-92b1-4b8b-8061-4b5732b2de40" containerName="main" Oct 30 02:33:57.343730 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.343701 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d99268-92b1-4b8b-8061-4b5732b2de40" containerName="main" Oct 30 02:33:57.343730 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.343715 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1d926a5c-800a-4b12-b134-be5c4ff7a619" containerName="istio-proxy" Oct 30 02:33:57.343730 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.343723 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d926a5c-800a-4b12-b134-be5c4ff7a619" containerName="istio-proxy" Oct 30 02:33:57.343910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.343753 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7e33e5cb-5be4-4ac8-ab50-506cac62ae3f" containerName="revoke-keys" Oct 30 02:33:57.343910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.343761 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e33e5cb-5be4-4ac8-ab50-506cac62ae3f" containerName="revoke-keys" Oct 30 02:33:57.343910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.343883 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b0f602-68d2-4494-a92f-c6df12bcdaad" containerName="main" Oct 30 02:33:57.343910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.343897 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d99268-92b1-4b8b-8061-4b5732b2de40" containerName="main" Oct 30 02:33:57.343910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.343907 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d926a5c-800a-4b12-b134-be5c4ff7a619" containerName="istio-proxy" Oct 30 02:33:57.344069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.343922 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e33e5cb-5be4-4ac8-ab50-506cac62ae3f" containerName="revoke-keys" Oct 30 02:33:57.348976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.348954 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx" Oct 30 02:33:57.365892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.365861 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx"] Oct 30 02:33:57.372128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.372085 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:33:57.372128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.372115 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:33:57.390266 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.390234 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4rq6\" (UniqueName: \"kubernetes.io/projected/ab218198-547c-419f-879f-675b090b1c67-kube-api-access-q4rq6\") pod \"maas-api-revoke-sub-5ee1fa15-ccbcx\" (UID: \"ab218198-547c-419f-879f-675b090b1c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx" Oct 30 02:33:57.390585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.390559 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab218198-547c-419f-879f-675b090b1c67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ee1fa15-ccbcx\" (UID: \"ab218198-547c-419f-879f-675b090b1c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx" Oct 30 02:33:57.390699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.390648 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab218198-547c-419f-879f-675b090b1c67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ee1fa15-ccbcx\" (UID: \"ab218198-547c-419f-879f-675b090b1c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx" Oct 30 02:33:57.399813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.399787 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:33:57.399955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.399889 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:33:57.491641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.491596 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab218198-547c-419f-879f-675b090b1c67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ee1fa15-ccbcx\" (UID: \"ab218198-547c-419f-879f-675b090b1c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx" Oct 30 02:33:57.491835 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.491696 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab218198-547c-419f-879f-675b090b1c67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ee1fa15-ccbcx\" (UID: \"ab218198-547c-419f-879f-675b090b1c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx" Oct 30 02:33:57.491948 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.491928 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4rq6\" (UniqueName: \"kubernetes.io/projected/ab218198-547c-419f-879f-675b090b1c67-kube-api-access-q4rq6\") pod \"maas-api-revoke-sub-5ee1fa15-ccbcx\" (UID: \"ab218198-547c-419f-879f-675b090b1c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx" Oct 30 02:33:57.492408 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.492385 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab218198-547c-419f-879f-675b090b1c67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ee1fa15-ccbcx\" (UID: \"ab218198-547c-419f-879f-675b090b1c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx" Oct 30 02:33:57.494043 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.494023 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab218198-547c-419f-879f-675b090b1c67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ee1fa15-ccbcx\" (UID: \"ab218198-547c-419f-879f-675b090b1c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx" Oct 30 02:33:57.514712 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.514687 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4rq6\" (UniqueName: \"kubernetes.io/projected/ab218198-547c-419f-879f-675b090b1c67-kube-api-access-q4rq6\") pod \"maas-api-revoke-sub-5ee1fa15-ccbcx\" (UID: \"ab218198-547c-419f-879f-675b090b1c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx" Oct 30 02:33:57.664124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.664044 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx" Oct 30 02:33:57.789636 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.789597 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:57.906050 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.905849 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:57.977599 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:57.977554 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx"] Oct 30 02:33:57.981173 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:33:57.981138 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab218198_547c_419f_879f_675b090b1c67.slice/crio-fad5486796eff33dc985b20dd31315d05c34790b5208319afdd694f150035fb1 WatchSource:0}: Error finding container fad5486796eff33dc985b20dd31315d05c34790b5208319afdd694f150035fb1: Status 404 returned error can't find the container with id fad5486796eff33dc985b20dd31315d05c34790b5208319afdd694f150035fb1 Oct 30 02:33:58.624727 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:58.624684 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx" event={"ID":"ab218198-547c-419f-879f-675b090b1c67","Type":"ContainerStarted","Data":"19421242744bb7d3a935abd1bb73e0021e1c13534250a5a2baf8de347a6fbfcf"} Oct 30 02:33:58.625170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:58.624752 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx" event={"ID":"ab218198-547c-419f-879f-675b090b1c67","Type":"ContainerStarted","Data":"fad5486796eff33dc985b20dd31315d05c34790b5208319afdd694f150035fb1"} Oct 30 02:33:58.668170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:58.668104 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx" podStartSLOduration=1.668067868 podStartE2EDuration="1.668067868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:33:58.661815217 +0000 UTC m=+3868.240606301" watchObservedRunningTime="2026-10-30 02:33:58.668067868 +0000 UTC m=+3868.246858966" Oct 30 02:33:59.531607 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:59.531570 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:33:59.640375 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:59.640340 5998 generic.go:356] "Generic (PLEG): container finished" podID="ab218198-547c-419f-879f-675b090b1c67" containerID="19421242744bb7d3a935abd1bb73e0021e1c13534250a5a2baf8de347a6fbfcf" exitCode=0 Oct 30 02:33:59.640866 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:33:59.640420 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx" event={"ID":"ab218198-547c-419f-879f-675b090b1c67","Type":"ContainerDied","Data":"19421242744bb7d3a935abd1bb73e0021e1c13534250a5a2baf8de347a6fbfcf"} Oct 30 02:34:00.823166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:00.823134 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx" Oct 30 02:34:00.850891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:00.850850 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab218198-547c-419f-879f-675b090b1c67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab218198-547c-419f-879f-675b090b1c67-maas-api-xks-ca\") pod \"ab218198-547c-419f-879f-675b090b1c67\" (UID: \"ab218198-547c-419f-879f-675b090b1c67\") " Oct 30 02:34:00.851127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:00.851005 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab218198-547c-419f-879f-675b090b1c67-kube-api-access-q4rq6\" (UniqueName: \"kubernetes.io/projected/ab218198-547c-419f-879f-675b090b1c67-kube-api-access-q4rq6\") pod \"ab218198-547c-419f-879f-675b090b1c67\" (UID: \"ab218198-547c-419f-879f-675b090b1c67\") " Oct 30 02:34:00.851127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:00.851066 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab218198-547c-419f-879f-675b090b1c67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab218198-547c-419f-879f-675b090b1c67-maas-api-openshift-ca\") pod \"ab218198-547c-419f-879f-675b090b1c67\" (UID: \"ab218198-547c-419f-879f-675b090b1c67\") " Oct 30 02:34:00.851672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:00.851640 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab218198-547c-419f-879f-675b090b1c67-maas-api-openshift-ca" pod "ab218198-547c-419f-879f-675b090b1c67" (UID: "ab218198-547c-419f-879f-675b090b1c67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:34:00.851962 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:00.851938 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab218198-547c-419f-879f-675b090b1c67-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:00.853265 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:00.853238 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab218198-547c-419f-879f-675b090b1c67-maas-api-xks-ca" pod "ab218198-547c-419f-879f-675b090b1c67" (UID: "ab218198-547c-419f-879f-675b090b1c67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:34:00.853357 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:00.853256 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab218198-547c-419f-879f-675b090b1c67-kube-api-access-q4rq6" pod "ab218198-547c-419f-879f-675b090b1c67" (UID: "ab218198-547c-419f-879f-675b090b1c67"). InnerVolumeSpecName "kube-api-access-q4rq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:34:00.953208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:00.953173 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab218198-547c-419f-879f-675b090b1c67-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:00.953208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:00.953207 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4rq6\" (UniqueName: \"kubernetes.io/projected/ab218198-547c-419f-879f-675b090b1c67-kube-api-access-q4rq6\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:01.207065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:01.206922 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:01.658363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:01.658273 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx" Oct 30 02:34:01.658363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:01.658295 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx" event={"ID":"ab218198-547c-419f-879f-675b090b1c67","Type":"ContainerDied","Data":"fad5486796eff33dc985b20dd31315d05c34790b5208319afdd694f150035fb1"} Oct 30 02:34:01.658363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:01.658345 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fad5486796eff33dc985b20dd31315d05c34790b5208319afdd694f150035fb1" Oct 30 02:34:02.897944 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:02.897911 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:04.607745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:04.607680 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:06.175128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:06.175026 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:07.784063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:07.784027 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:08.205072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.205039 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz"] Oct 30 02:34:08.254885 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.254848 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:08.795517 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.795475 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx"] Oct 30 02:34:08.796399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.796371 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ab218198-547c-419f-879f-675b090b1c67" containerName="revoke-keys" Oct 30 02:34:08.796399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.796399 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab218198-547c-419f-879f-675b090b1c67" containerName="revoke-keys" Oct 30 02:34:08.796653 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.796637 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab218198-547c-419f-879f-675b090b1c67" containerName="revoke-keys" Oct 30 02:34:08.799847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.799822 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx" Oct 30 02:34:08.831367 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.831334 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:34:08.831519 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.831373 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:34:08.837521 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.837490 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:34:08.837521 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.837505 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:34:08.844291 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.844260 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx"] Oct 30 02:34:08.885882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.885845 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64ea060-92e7-4058-a390-f9192c1c879b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7aa71fe-lmtqx\" (UID: \"b64ea060-92e7-4058-a390-f9192c1c879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx" Oct 30 02:34:08.886107 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.885891 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl29x\" (UniqueName: \"kubernetes.io/projected/b64ea060-92e7-4058-a390-f9192c1c879b-kube-api-access-zl29x\") pod \"maas-api-revoke-sub-e7aa71fe-lmtqx\" (UID: \"b64ea060-92e7-4058-a390-f9192c1c879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx" Oct 30 02:34:08.886187 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.886085 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64ea060-92e7-4058-a390-f9192c1c879b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7aa71fe-lmtqx\" (UID: \"b64ea060-92e7-4058-a390-f9192c1c879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx" Oct 30 02:34:08.987211 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.987163 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64ea060-92e7-4058-a390-f9192c1c879b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7aa71fe-lmtqx\" (UID: \"b64ea060-92e7-4058-a390-f9192c1c879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx" Oct 30 02:34:08.987429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.987407 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64ea060-92e7-4058-a390-f9192c1c879b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7aa71fe-lmtqx\" (UID: \"b64ea060-92e7-4058-a390-f9192c1c879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx" Oct 30 02:34:08.987507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.987440 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl29x\" (UniqueName: \"kubernetes.io/projected/b64ea060-92e7-4058-a390-f9192c1c879b-kube-api-access-zl29x\") pod \"maas-api-revoke-sub-e7aa71fe-lmtqx\" (UID: \"b64ea060-92e7-4058-a390-f9192c1c879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx" Oct 30 02:34:08.987871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.987847 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64ea060-92e7-4058-a390-f9192c1c879b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7aa71fe-lmtqx\" (UID: \"b64ea060-92e7-4058-a390-f9192c1c879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx" Oct 30 02:34:08.989833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:08.989805 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64ea060-92e7-4058-a390-f9192c1c879b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7aa71fe-lmtqx\" (UID: \"b64ea060-92e7-4058-a390-f9192c1c879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx" Oct 30 02:34:09.013362 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:09.013314 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl29x\" (UniqueName: \"kubernetes.io/projected/b64ea060-92e7-4058-a390-f9192c1c879b-kube-api-access-zl29x\") pod \"maas-api-revoke-sub-e7aa71fe-lmtqx\" (UID: \"b64ea060-92e7-4058-a390-f9192c1c879b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx" Oct 30 02:34:09.118918 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:09.118815 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx" Oct 30 02:34:09.429014 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:09.428973 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:09.430289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:09.430231 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx"] Oct 30 02:34:09.433965 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:34:09.433937 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64ea060_92e7_4058_a390_f9192c1c879b.slice/crio-734cf640de19f10b7ef0a0dd12f760e833de857960de2be0a8a471e117804fcf WatchSource:0}: Error finding container 734cf640de19f10b7ef0a0dd12f760e833de857960de2be0a8a471e117804fcf: Status 404 returned error can't find the container with id 734cf640de19f10b7ef0a0dd12f760e833de857960de2be0a8a471e117804fcf Oct 30 02:34:09.727440 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:09.727396 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx" event={"ID":"b64ea060-92e7-4058-a390-f9192c1c879b","Type":"ContainerStarted","Data":"0c35026dceaa208d7b499a98bd5c0bf5fe40644416287b485d4b2c86224ff2c4"} Oct 30 02:34:09.727632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:09.727464 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx" event={"ID":"b64ea060-92e7-4058-a390-f9192c1c879b","Type":"ContainerStarted","Data":"734cf640de19f10b7ef0a0dd12f760e833de857960de2be0a8a471e117804fcf"} Oct 30 02:34:09.768326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:09.768262 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx" podStartSLOduration=1.768243933 podStartE2EDuration="1.768243933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:34:09.766943633 +0000 UTC m=+3879.345734713" watchObservedRunningTime="2026-10-30 02:34:09.768243933 +0000 UTC m=+3879.347035018" Oct 30 02:34:10.740062 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:10.740022 5998 generic.go:356] "Generic (PLEG): container finished" podID="b64ea060-92e7-4058-a390-f9192c1c879b" containerID="0c35026dceaa208d7b499a98bd5c0bf5fe40644416287b485d4b2c86224ff2c4" exitCode=0 Oct 30 02:34:10.740538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:10.740148 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx" event={"ID":"b64ea060-92e7-4058-a390-f9192c1c879b","Type":"ContainerDied","Data":"0c35026dceaa208d7b499a98bd5c0bf5fe40644416287b485d4b2c86224ff2c4"} Oct 30 02:34:11.087404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:11.087322 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:11.930732 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:11.930352 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx" Oct 30 02:34:11.943801 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:11.943761 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64ea060-92e7-4058-a390-f9192c1c879b-kube-api-access-zl29x\" (UniqueName: \"kubernetes.io/projected/b64ea060-92e7-4058-a390-f9192c1c879b-kube-api-access-zl29x\") pod \"b64ea060-92e7-4058-a390-f9192c1c879b\" (UID: \"b64ea060-92e7-4058-a390-f9192c1c879b\") " Oct 30 02:34:11.944064 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:11.944039 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b64ea060-92e7-4058-a390-f9192c1c879b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64ea060-92e7-4058-a390-f9192c1c879b-maas-api-xks-ca\") pod \"b64ea060-92e7-4058-a390-f9192c1c879b\" (UID: \"b64ea060-92e7-4058-a390-f9192c1c879b\") " Oct 30 02:34:11.944306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:11.944285 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64ea060-92e7-4058-a390-f9192c1c879b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64ea060-92e7-4058-a390-f9192c1c879b-maas-api-openshift-ca\") pod \"b64ea060-92e7-4058-a390-f9192c1c879b\" (UID: \"b64ea060-92e7-4058-a390-f9192c1c879b\") " Oct 30 02:34:11.944716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:11.944686 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64ea060-92e7-4058-a390-f9192c1c879b-maas-api-openshift-ca" pod "b64ea060-92e7-4058-a390-f9192c1c879b" (UID: "b64ea060-92e7-4058-a390-f9192c1c879b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:34:11.945307 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:11.945279 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b64ea060-92e7-4058-a390-f9192c1c879b-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:11.946344 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:11.946318 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b64ea060-92e7-4058-a390-f9192c1c879b-maas-api-xks-ca" pod "b64ea060-92e7-4058-a390-f9192c1c879b" (UID: "b64ea060-92e7-4058-a390-f9192c1c879b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:34:11.946561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:11.946543 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64ea060-92e7-4058-a390-f9192c1c879b-kube-api-access-zl29x" pod "b64ea060-92e7-4058-a390-f9192c1c879b" (UID: "b64ea060-92e7-4058-a390-f9192c1c879b"). InnerVolumeSpecName "kube-api-access-zl29x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:34:12.047889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:12.046821 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl29x\" (UniqueName: \"kubernetes.io/projected/b64ea060-92e7-4058-a390-f9192c1c879b-kube-api-access-zl29x\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:12.047889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:12.046860 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b64ea060-92e7-4058-a390-f9192c1c879b-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:12.612426 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:12.612380 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:12.758339 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:12.758299 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx" Oct 30 02:34:12.758339 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:12.758326 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx" event={"ID":"b64ea060-92e7-4058-a390-f9192c1c879b","Type":"ContainerDied","Data":"734cf640de19f10b7ef0a0dd12f760e833de857960de2be0a8a471e117804fcf"} Oct 30 02:34:12.758582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:12.758393 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="734cf640de19f10b7ef0a0dd12f760e833de857960de2be0a8a471e117804fcf" Oct 30 02:34:14.211469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:14.211429 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:15.869706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:15.869652 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:17.445285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:17.445210 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:19.056960 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:19.056925 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:19.973236 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:19.973196 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:34:20.789650 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:20.789610 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:22.484409 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:22.484372 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:23.253958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.253912 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" podUID="4d6bd9d9-a6d9-4f8e-93ea-db253f208e02" containerName="main" containerID="cri-o://d77e875a5dc67dc475632a08d74e48df98b95ef8526ea8e482140526645884a4" gracePeriod=45 Oct 30 02:34:23.575288 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.575262 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:34:23.654445 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.654408 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-kube-api-access-86ng9\" (UniqueName: \"kubernetes.io/projected/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-kube-api-access-86ng9\") pod \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " Oct 30 02:34:23.654660 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.654484 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-model-cache\") pod \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " Oct 30 02:34:23.654660 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.654569 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-tmp-dir\") pod \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " Oct 30 02:34:23.654660 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.654645 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-tls-certs\" (UniqueName: \"kubernetes.io/secret/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-tls-certs\") pod \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " Oct 30 02:34:23.654857 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.654684 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-home\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-home\") pod \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " Oct 30 02:34:23.654857 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.654800 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-dshm\") pod \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\" (UID: \"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02\") " Oct 30 02:34:23.655389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.655067 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-tmp-dir" pod "4d6bd9d9-a6d9-4f8e-93ea-db253f208e02" (UID: "4d6bd9d9-a6d9-4f8e-93ea-db253f208e02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:34:23.655389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.655301 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-model-cache" pod "4d6bd9d9-a6d9-4f8e-93ea-db253f208e02" (UID: "4d6bd9d9-a6d9-4f8e-93ea-db253f208e02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:34:23.655790 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.655691 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:23.655790 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.655737 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:23.656021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.655982 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-home" pod "4d6bd9d9-a6d9-4f8e-93ea-db253f208e02" (UID: "4d6bd9d9-a6d9-4f8e-93ea-db253f208e02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:34:23.658319 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.658294 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-dshm" pod "4d6bd9d9-a6d9-4f8e-93ea-db253f208e02" (UID: "4d6bd9d9-a6d9-4f8e-93ea-db253f208e02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:34:23.658425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.658315 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-kube-api-access-86ng9" pod "4d6bd9d9-a6d9-4f8e-93ea-db253f208e02" (UID: "4d6bd9d9-a6d9-4f8e-93ea-db253f208e02"). InnerVolumeSpecName "kube-api-access-86ng9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:34:23.658584 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.658552 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-tls-certs" pod "4d6bd9d9-a6d9-4f8e-93ea-db253f208e02" (UID: "4d6bd9d9-a6d9-4f8e-93ea-db253f208e02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:34:23.757734 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.757693 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:23.757734 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.757730 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86ng9\" (UniqueName: \"kubernetes.io/projected/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-kube-api-access-86ng9\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:23.757734 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.757742 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:23.758003 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.757752 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:23.948376 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.948335 5998 generic.go:356] "Generic (PLEG): container finished" podID="4d6bd9d9-a6d9-4f8e-93ea-db253f208e02" containerID="d77e875a5dc67dc475632a08d74e48df98b95ef8526ea8e482140526645884a4" exitCode=0 Oct 30 02:34:23.948592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.948454 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" Oct 30 02:34:23.948592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.948484 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" event={"ID":"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02","Type":"ContainerDied","Data":"d77e875a5dc67dc475632a08d74e48df98b95ef8526ea8e482140526645884a4"} Oct 30 02:34:23.948592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.948530 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz" event={"ID":"4d6bd9d9-a6d9-4f8e-93ea-db253f208e02","Type":"ContainerDied","Data":"313959391aea5179164453a843c78b62584294d5e7d2d661530af9c7aa490e9c"} Oct 30 02:34:23.948592 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.948560 5998 scope.go:122] "RemoveContainer" containerID="d77e875a5dc67dc475632a08d74e48df98b95ef8526ea8e482140526645884a4" Oct 30 02:34:23.960436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.960410 5998 scope.go:122] "RemoveContainer" containerID="d77e875a5dc67dc475632a08d74e48df98b95ef8526ea8e482140526645884a4" Oct 30 02:34:23.960711 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:34:23.960685 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d77e875a5dc67dc475632a08d74e48df98b95ef8526ea8e482140526645884a4\": container with ID starting with d77e875a5dc67dc475632a08d74e48df98b95ef8526ea8e482140526645884a4 not found: ID does not exist" containerID="d77e875a5dc67dc475632a08d74e48df98b95ef8526ea8e482140526645884a4" Oct 30 02:34:23.960816 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.960718 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d77e875a5dc67dc475632a08d74e48df98b95ef8526ea8e482140526645884a4"} err="failed to get container status \"d77e875a5dc67dc475632a08d74e48df98b95ef8526ea8e482140526645884a4\": rpc error: code = NotFound desc = could not find container \"d77e875a5dc67dc475632a08d74e48df98b95ef8526ea8e482140526645884a4\": container with ID starting with d77e875a5dc67dc475632a08d74e48df98b95ef8526ea8e482140526645884a4 not found: ID does not exist" Oct 30 02:34:23.996852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:23.996808 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz"] Oct 30 02:34:24.007152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:24.007118 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2943ff/test-model-2943ff-kserve-9c9949c6c-kwfvz"] Oct 30 02:34:24.155174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:24.155130 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:24.542359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:24.542320 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d6bd9d9-a6d9-4f8e-93ea-db253f208e02" path="/var/lib/kubelet/pods/4d6bd9d9-a6d9-4f8e-93ea-db253f208e02/volumes" Oct 30 02:34:24.857155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:24.857050 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:25.225752 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:34:25.225709 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:34:25.685991 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:25.685952 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:25.828340 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:25.828287 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:25.942831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:25.942722 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:27.196971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:27.196937 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:28.761632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:28.761595 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:28.888264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:28.883659 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:29.203232 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:29.203193 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:30.218200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:30.218163 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:30.322406 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:30.322359 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:30.486870 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:30.486794 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:32.220995 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:32.220911 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:33.849054 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:33.849014 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:34.276627 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:34.276592 5998 scope.go:122] "RemoveContainer" containerID="90a67e29b23d6d65049494bbcc8b7c67dec1df6e5be8193599a4a337df17cbe4" Oct 30 02:34:34.288508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:34.288481 5998 scope.go:122] "RemoveContainer" containerID="db04f6788fc8f5d2b78247639256e064fba851a390ddf7257165263252464949" Oct 30 02:34:35.511565 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:35.511519 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:36.345294 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:36.345249 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:36.481892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:36.481855 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:36.607424 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:36.606689 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:37.328389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:37.328321 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:39.030681 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:39.030644 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:40.232406 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:40.232361 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:40.549687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:40.549602 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:41.411772 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:41.411729 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:42.718762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:42.718712 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:43.205113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:43.201509 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:43.326712 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:43.326673 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:43.470861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:43.470768 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:43.911501 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:43.911444 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:44.340355 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:44.340211 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999"] Oct 30 02:34:44.342078 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:44.342040 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" podUID="bf65846d-6629-424c-9eda-4c359fd1992a" containerName="istio-proxy" containerID="cri-o://3c7d4f5f4edb8be2cc0aabd86366460c6945492f4a4e5471a0178e8b585da5fa" gracePeriod=30 Oct 30 02:34:44.355330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:44.355268 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:45.153012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:45.152951 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:45.290137 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:45.290104 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:45.403288 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:45.403196 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:45.537117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:45.530377 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:46.076834 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.076802 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:46.247947 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.247910 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4"] Oct 30 02:34:46.248925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.248903 5998 cpu_manager.go:414] "containerMap: removing container" podUID="b64ea060-92e7-4058-a390-f9192c1c879b" containerName="revoke-keys" Oct 30 02:34:46.248977 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.248930 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64ea060-92e7-4058-a390-f9192c1c879b" containerName="revoke-keys" Oct 30 02:34:46.249015 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.248994 5998 cpu_manager.go:414] "containerMap: removing container" podUID="4d6bd9d9-a6d9-4f8e-93ea-db253f208e02" containerName="main" Oct 30 02:34:46.249015 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.249002 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6bd9d9-a6d9-4f8e-93ea-db253f208e02" containerName="main" Oct 30 02:34:46.249171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.249157 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6bd9d9-a6d9-4f8e-93ea-db253f208e02" containerName="main" Oct 30 02:34:46.249217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.249180 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64ea060-92e7-4058-a390-f9192c1c879b" containerName="revoke-keys" Oct 30 02:34:46.253083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.253059 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4" Oct 30 02:34:46.275398 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.275364 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4"] Oct 30 02:34:46.281936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.281903 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:34:46.282100 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.281932 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:34:46.282100 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.281932 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:34:46.284538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.284515 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:34:46.332540 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.332436 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0443d77-6zkh4\" (UID: \"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4" Oct 30 02:34:46.332727 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.332548 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0443d77-6zkh4\" (UID: \"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4" Oct 30 02:34:46.332727 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.332637 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8p9h\" (UniqueName: \"kubernetes.io/projected/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-kube-api-access-d8p9h\") pod \"maas-api-revoke-sub-f0443d77-6zkh4\" (UID: \"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4" Oct 30 02:34:46.434865 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.434815 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0443d77-6zkh4\" (UID: \"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4" Oct 30 02:34:46.435194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.435167 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0443d77-6zkh4\" (UID: \"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4" Oct 30 02:34:46.435416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.435399 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8p9h\" (UniqueName: \"kubernetes.io/projected/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-kube-api-access-d8p9h\") pod \"maas-api-revoke-sub-f0443d77-6zkh4\" (UID: \"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4" Oct 30 02:34:46.437629 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.437606 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0443d77-6zkh4\" (UID: \"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4" Oct 30 02:34:46.455641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.455595 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0443d77-6zkh4\" (UID: \"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4" Oct 30 02:34:46.461046 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.461019 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:46.463365 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.463346 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8p9h\" (UniqueName: \"kubernetes.io/projected/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-kube-api-access-d8p9h\") pod \"maas-api-revoke-sub-f0443d77-6zkh4\" (UID: \"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4" Oct 30 02:34:46.466625 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.466599 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj"] Oct 30 02:34:46.466870 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.466838 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" podUID="3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" containerName="istio-proxy" containerID="cri-o://6b484ea0b153eeb2afe624c009a9e4151c5563e55616c9a89173992d840e1a45" gracePeriod=30 Oct 30 02:34:46.515313 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.515275 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6"] Oct 30 02:34:46.520354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.520328 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6" Oct 30 02:34:46.579804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.573376 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4" Oct 30 02:34:46.691113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.687148 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/650423de-a897-4e26-b2a8-c822c8642895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e30d778-s6wt6\" (UID: \"650423de-a897-4e26-b2a8-c822c8642895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6" Oct 30 02:34:46.691113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.687229 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w85cp\" (UniqueName: \"kubernetes.io/projected/650423de-a897-4e26-b2a8-c822c8642895-kube-api-access-w85cp\") pod \"maas-api-revoke-sub-0e30d778-s6wt6\" (UID: \"650423de-a897-4e26-b2a8-c822c8642895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6" Oct 30 02:34:46.691113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.687536 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/650423de-a897-4e26-b2a8-c822c8642895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e30d778-s6wt6\" (UID: \"650423de-a897-4e26-b2a8-c822c8642895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6" Oct 30 02:34:46.795950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.787879 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6"] Oct 30 02:34:46.795950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.789065 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/650423de-a897-4e26-b2a8-c822c8642895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e30d778-s6wt6\" (UID: \"650423de-a897-4e26-b2a8-c822c8642895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6" Oct 30 02:34:46.795950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.789221 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/650423de-a897-4e26-b2a8-c822c8642895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e30d778-s6wt6\" (UID: \"650423de-a897-4e26-b2a8-c822c8642895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6" Oct 30 02:34:46.795950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.789282 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w85cp\" (UniqueName: \"kubernetes.io/projected/650423de-a897-4e26-b2a8-c822c8642895-kube-api-access-w85cp\") pod \"maas-api-revoke-sub-0e30d778-s6wt6\" (UID: \"650423de-a897-4e26-b2a8-c822c8642895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6" Oct 30 02:34:46.795950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.791514 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/650423de-a897-4e26-b2a8-c822c8642895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e30d778-s6wt6\" (UID: \"650423de-a897-4e26-b2a8-c822c8642895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6" Oct 30 02:34:46.801178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.799014 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/650423de-a897-4e26-b2a8-c822c8642895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e30d778-s6wt6\" (UID: \"650423de-a897-4e26-b2a8-c822c8642895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6" Oct 30 02:34:46.814801 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.814759 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w85cp\" (UniqueName: \"kubernetes.io/projected/650423de-a897-4e26-b2a8-c822c8642895-kube-api-access-w85cp\") pod \"maas-api-revoke-sub-0e30d778-s6wt6\" (UID: \"650423de-a897-4e26-b2a8-c822c8642895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6" Oct 30 02:34:46.840194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:46.836424 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6" Oct 30 02:34:47.153162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:47.153069 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4"] Oct 30 02:34:47.168242 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:34:47.168210 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76fafcf_a716_4a3b_86fb_a12a17f1ae5e.slice/crio-f44397b7da65b5abfb85fae0d36c2e2c326ef6364d0cf82538f18d6c53e823f3 WatchSource:0}: Error finding container f44397b7da65b5abfb85fae0d36c2e2c326ef6364d0cf82538f18d6c53e823f3: Status 404 returned error can't find the container with id f44397b7da65b5abfb85fae0d36c2e2c326ef6364d0cf82538f18d6c53e823f3 Oct 30 02:34:47.182238 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:47.182204 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6"] Oct 30 02:34:47.184521 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:34:47.184492 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod650423de_a897_4e26_b2a8_c822c8642895.slice/crio-59bc17172e1fb02e2cf6343b73f7aaf9c53d563e469840dd70373f78943ecfae WatchSource:0}: Error finding container 59bc17172e1fb02e2cf6343b73f7aaf9c53d563e469840dd70373f78943ecfae: Status 404 returned error can't find the container with id 59bc17172e1fb02e2cf6343b73f7aaf9c53d563e469840dd70373f78943ecfae Oct 30 02:34:47.268871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:47.268834 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4" event={"ID":"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e","Type":"ContainerStarted","Data":"f44397b7da65b5abfb85fae0d36c2e2c326ef6364d0cf82538f18d6c53e823f3"} Oct 30 02:34:47.270291 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:47.270248 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6" event={"ID":"650423de-a897-4e26-b2a8-c822c8642895","Type":"ContainerStarted","Data":"59bc17172e1fb02e2cf6343b73f7aaf9c53d563e469840dd70373f78943ecfae"} Oct 30 02:34:47.390822 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:47.390780 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:47.553979 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:47.553946 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:47.738754 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:47.738714 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:47.879716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:47.879682 5998 patch_prober.go:29] interesting pod/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:34:47.879918 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:47.879735 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" podUID="bf65846d-6629-424c-9eda-4c359fd1992a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:34:48.280602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:48.280560 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4" event={"ID":"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e","Type":"ContainerStarted","Data":"58a6254e95a0af3ab30624c257380d9079bd470d2e6f90ae506ec047b1f3dd30"} Oct 30 02:34:48.282414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:48.282382 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6" event={"ID":"650423de-a897-4e26-b2a8-c822c8642895","Type":"ContainerStarted","Data":"89b2eb57d9d808784e44714c8cf40cd787bbdf56cfe0e9bb7ba6be8a6db62cde"} Oct 30 02:34:48.337028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:48.336865 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4" podStartSLOduration=2.336844603 podStartE2EDuration="2.336844603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:34:48.334379161 +0000 UTC m=+3917.913170244" watchObservedRunningTime="2026-10-30 02:34:48.336844603 +0000 UTC m=+3917.915635684" Oct 30 02:34:48.374175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:48.374120 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6" podStartSLOduration=2.37410137 podStartE2EDuration="2.37410137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:34:48.371750156 +0000 UTC m=+3917.950541236" watchObservedRunningTime="2026-10-30 02:34:48.37410137 +0000 UTC m=+3917.952892449" Oct 30 02:34:49.292806 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.292766 5998 generic.go:356] "Generic (PLEG): container finished" podID="a76fafcf-a716-4a3b-86fb-a12a17f1ae5e" containerID="58a6254e95a0af3ab30624c257380d9079bd470d2e6f90ae506ec047b1f3dd30" exitCode=0 Oct 30 02:34:49.293299 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.292852 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4" event={"ID":"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e","Type":"ContainerDied","Data":"58a6254e95a0af3ab30624c257380d9079bd470d2e6f90ae506ec047b1f3dd30"} Oct 30 02:34:49.294717 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.294690 5998 generic.go:356] "Generic (PLEG): container finished" podID="650423de-a897-4e26-b2a8-c822c8642895" containerID="89b2eb57d9d808784e44714c8cf40cd787bbdf56cfe0e9bb7ba6be8a6db62cde" exitCode=0 Oct 30 02:34:49.294847 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.294771 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6" event={"ID":"650423de-a897-4e26-b2a8-c822c8642895","Type":"ContainerDied","Data":"89b2eb57d9d808784e44714c8cf40cd787bbdf56cfe0e9bb7ba6be8a6db62cde"} Oct 30 02:34:49.399005 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.398968 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:49.691390 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.691355 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:34:49.747432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.747388 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-istio-envoy\") pod \"bf65846d-6629-424c-9eda-4c359fd1992a\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " Oct 30 02:34:49.747617 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.747507 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-workload-socket\") pod \"bf65846d-6629-424c-9eda-4c359fd1992a\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " Oct 30 02:34:49.747617 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.747584 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf65846d-6629-424c-9eda-4c359fd1992a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf65846d-6629-424c-9eda-4c359fd1992a-istiod-ca-cert\") pod \"bf65846d-6629-424c-9eda-4c359fd1992a\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " Oct 30 02:34:49.747767 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.747658 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-istio-data\") pod \"bf65846d-6629-424c-9eda-4c359fd1992a\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " Oct 30 02:34:49.747767 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.747716 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf65846d-6629-424c-9eda-4c359fd1992a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf65846d-6629-424c-9eda-4c359fd1992a-istio-podinfo\") pod \"bf65846d-6629-424c-9eda-4c359fd1992a\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " Oct 30 02:34:49.747891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.747807 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf65846d-6629-424c-9eda-4c359fd1992a-istio-token\" (UniqueName: \"kubernetes.io/projected/bf65846d-6629-424c-9eda-4c359fd1992a-istio-token\") pod \"bf65846d-6629-424c-9eda-4c359fd1992a\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " Oct 30 02:34:49.747891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.747863 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-workload-certs\") pod \"bf65846d-6629-424c-9eda-4c359fd1992a\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " Oct 30 02:34:49.748012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.747898 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf65846d-6629-424c-9eda-4c359fd1992a-kube-api-access-p7kv4\" (UniqueName: \"kubernetes.io/projected/bf65846d-6629-424c-9eda-4c359fd1992a-kube-api-access-p7kv4\") pod \"bf65846d-6629-424c-9eda-4c359fd1992a\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " Oct 30 02:34:49.748012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.747947 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-credential-socket\") pod \"bf65846d-6629-424c-9eda-4c359fd1992a\" (UID: \"bf65846d-6629-424c-9eda-4c359fd1992a\") " Oct 30 02:34:49.750243 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.749398 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-workload-socket" pod "bf65846d-6629-424c-9eda-4c359fd1992a" (UID: "bf65846d-6629-424c-9eda-4c359fd1992a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:34:49.750243 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.749780 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf65846d-6629-424c-9eda-4c359fd1992a-istiod-ca-cert" pod "bf65846d-6629-424c-9eda-4c359fd1992a" (UID: "bf65846d-6629-424c-9eda-4c359fd1992a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:34:49.750243 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.750051 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-workload-certs" pod "bf65846d-6629-424c-9eda-4c359fd1992a" (UID: "bf65846d-6629-424c-9eda-4c359fd1992a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:34:49.750599 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.750431 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-credential-socket" pod "bf65846d-6629-424c-9eda-4c359fd1992a" (UID: "bf65846d-6629-424c-9eda-4c359fd1992a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:34:49.750691 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.750605 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-istio-data" pod "bf65846d-6629-424c-9eda-4c359fd1992a" (UID: "bf65846d-6629-424c-9eda-4c359fd1992a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:34:49.755618 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.755587 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf65846d-6629-424c-9eda-4c359fd1992a-kube-api-access-p7kv4" pod "bf65846d-6629-424c-9eda-4c359fd1992a" (UID: "bf65846d-6629-424c-9eda-4c359fd1992a"). InnerVolumeSpecName "kube-api-access-p7kv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:34:49.756048 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.756008 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf65846d-6629-424c-9eda-4c359fd1992a-istio-token" pod "bf65846d-6629-424c-9eda-4c359fd1992a" (UID: "bf65846d-6629-424c-9eda-4c359fd1992a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:34:49.761579 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.761544 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf65846d-6629-424c-9eda-4c359fd1992a-istio-podinfo" pod "bf65846d-6629-424c-9eda-4c359fd1992a" (UID: "bf65846d-6629-424c-9eda-4c359fd1992a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:34:49.765239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.765200 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-istio-envoy" pod "bf65846d-6629-424c-9eda-4c359fd1992a" (UID: "bf65846d-6629-424c-9eda-4c359fd1992a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:34:49.850336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.850241 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:49.850336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.850279 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7kv4\" (UniqueName: \"kubernetes.io/projected/bf65846d-6629-424c-9eda-4c359fd1992a-kube-api-access-p7kv4\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:49.850336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.850294 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:49.850336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.850308 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:49.850336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.850320 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:49.850336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.850332 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf65846d-6629-424c-9eda-4c359fd1992a-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:49.850694 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.850343 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf65846d-6629-424c-9eda-4c359fd1992a-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:49.850694 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.850355 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf65846d-6629-424c-9eda-4c359fd1992a-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:49.850694 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:49.850374 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf65846d-6629-424c-9eda-4c359fd1992a-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:50.305536 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.305490 5998 generic.go:356] "Generic (PLEG): container finished" podID="bf65846d-6629-424c-9eda-4c359fd1992a" containerID="3c7d4f5f4edb8be2cc0aabd86366460c6945492f4a4e5471a0178e8b585da5fa" exitCode=0 Oct 30 02:34:50.305536 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.305536 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" Oct 30 02:34:50.306160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.305575 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" event={"ID":"bf65846d-6629-424c-9eda-4c359fd1992a","Type":"ContainerDied","Data":"3c7d4f5f4edb8be2cc0aabd86366460c6945492f4a4e5471a0178e8b585da5fa"} Oct 30 02:34:50.306160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.305632 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999" event={"ID":"bf65846d-6629-424c-9eda-4c359fd1992a","Type":"ContainerDied","Data":"99295b5c40a88c10aebfec2825aed110deedcacdc2748b0f8dc6a062b234bc32"} Oct 30 02:34:50.306160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.305663 5998 scope.go:122] "RemoveContainer" containerID="3c7d4f5f4edb8be2cc0aabd86366460c6945492f4a4e5471a0178e8b585da5fa" Oct 30 02:34:50.320436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.320410 5998 scope.go:122] "RemoveContainer" containerID="8ee63260b4416374d92f1a6eb0f58f9d8fb329fb304a04685a8d2622b454fca2" Oct 30 02:34:50.335618 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.335585 5998 scope.go:122] "RemoveContainer" containerID="3c7d4f5f4edb8be2cc0aabd86366460c6945492f4a4e5471a0178e8b585da5fa" Oct 30 02:34:50.336061 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:34:50.336018 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c7d4f5f4edb8be2cc0aabd86366460c6945492f4a4e5471a0178e8b585da5fa\": container with ID starting with 3c7d4f5f4edb8be2cc0aabd86366460c6945492f4a4e5471a0178e8b585da5fa not found: ID does not exist" containerID="3c7d4f5f4edb8be2cc0aabd86366460c6945492f4a4e5471a0178e8b585da5fa" Oct 30 02:34:50.336747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.336072 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7d4f5f4edb8be2cc0aabd86366460c6945492f4a4e5471a0178e8b585da5fa"} err="failed to get container status \"3c7d4f5f4edb8be2cc0aabd86366460c6945492f4a4e5471a0178e8b585da5fa\": rpc error: code = NotFound desc = could not find container \"3c7d4f5f4edb8be2cc0aabd86366460c6945492f4a4e5471a0178e8b585da5fa\": container with ID starting with 3c7d4f5f4edb8be2cc0aabd86366460c6945492f4a4e5471a0178e8b585da5fa not found: ID does not exist" Oct 30 02:34:50.336747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.336110 5998 scope.go:122] "RemoveContainer" containerID="8ee63260b4416374d92f1a6eb0f58f9d8fb329fb304a04685a8d2622b454fca2" Oct 30 02:34:50.336747 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:34:50.336433 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee63260b4416374d92f1a6eb0f58f9d8fb329fb304a04685a8d2622b454fca2\": container with ID starting with 8ee63260b4416374d92f1a6eb0f58f9d8fb329fb304a04685a8d2622b454fca2 not found: ID does not exist" containerID="8ee63260b4416374d92f1a6eb0f58f9d8fb329fb304a04685a8d2622b454fca2" Oct 30 02:34:50.336747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.336476 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee63260b4416374d92f1a6eb0f58f9d8fb329fb304a04685a8d2622b454fca2"} err="failed to get container status \"8ee63260b4416374d92f1a6eb0f58f9d8fb329fb304a04685a8d2622b454fca2\": rpc error: code = NotFound desc = could not find container \"8ee63260b4416374d92f1a6eb0f58f9d8fb329fb304a04685a8d2622b454fca2\": container with ID starting with 8ee63260b4416374d92f1a6eb0f58f9d8fb329fb304a04685a8d2622b454fca2 not found: ID does not exist" Oct 30 02:34:50.391935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.391862 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999"] Oct 30 02:34:50.400246 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.398876 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2943ff-openshift-default-6466bddbf9-c8999"] Oct 30 02:34:50.493793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.493758 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4" Oct 30 02:34:50.540704 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.540676 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6" Oct 30 02:34:50.543868 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.543837 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf65846d-6629-424c-9eda-4c359fd1992a" path="/var/lib/kubelet/pods/bf65846d-6629-424c-9eda-4c359fd1992a/volumes" Oct 30 02:34:50.559615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.559542 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-maas-api-openshift-ca\") pod \"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e\" (UID: \"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e\") " Oct 30 02:34:50.559775 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.559624 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-maas-api-xks-ca\") pod \"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e\" (UID: \"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e\") " Oct 30 02:34:50.559775 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.559721 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/650423de-a897-4e26-b2a8-c822c8642895-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/650423de-a897-4e26-b2a8-c822c8642895-maas-api-xks-ca\") pod \"650423de-a897-4e26-b2a8-c822c8642895\" (UID: \"650423de-a897-4e26-b2a8-c822c8642895\") " Oct 30 02:34:50.559775 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.559755 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/650423de-a897-4e26-b2a8-c822c8642895-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/650423de-a897-4e26-b2a8-c822c8642895-maas-api-openshift-ca\") pod \"650423de-a897-4e26-b2a8-c822c8642895\" (UID: \"650423de-a897-4e26-b2a8-c822c8642895\") " Oct 30 02:34:50.559968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.559858 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/650423de-a897-4e26-b2a8-c822c8642895-kube-api-access-w85cp\" (UniqueName: \"kubernetes.io/projected/650423de-a897-4e26-b2a8-c822c8642895-kube-api-access-w85cp\") pod \"650423de-a897-4e26-b2a8-c822c8642895\" (UID: \"650423de-a897-4e26-b2a8-c822c8642895\") " Oct 30 02:34:50.560106 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.559972 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-kube-api-access-d8p9h\" (UniqueName: \"kubernetes.io/projected/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-kube-api-access-d8p9h\") pod \"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e\" (UID: \"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e\") " Oct 30 02:34:50.560106 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.559979 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-maas-api-openshift-ca" pod "a76fafcf-a716-4a3b-86fb-a12a17f1ae5e" (UID: "a76fafcf-a716-4a3b-86fb-a12a17f1ae5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:34:50.561832 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.560364 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/650423de-a897-4e26-b2a8-c822c8642895-maas-api-openshift-ca" pod "650423de-a897-4e26-b2a8-c822c8642895" (UID: "650423de-a897-4e26-b2a8-c822c8642895"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:34:50.561832 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.560610 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/650423de-a897-4e26-b2a8-c822c8642895-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:50.561832 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.560632 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:50.562789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.562729 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/650423de-a897-4e26-b2a8-c822c8642895-maas-api-xks-ca" pod "650423de-a897-4e26-b2a8-c822c8642895" (UID: "650423de-a897-4e26-b2a8-c822c8642895"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:34:50.562789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.562738 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-maas-api-xks-ca" pod "a76fafcf-a716-4a3b-86fb-a12a17f1ae5e" (UID: "a76fafcf-a716-4a3b-86fb-a12a17f1ae5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:34:50.562789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.562756 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/650423de-a897-4e26-b2a8-c822c8642895-kube-api-access-w85cp" pod "650423de-a897-4e26-b2a8-c822c8642895" (UID: "650423de-a897-4e26-b2a8-c822c8642895"). InnerVolumeSpecName "kube-api-access-w85cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:34:50.564044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.564012 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-kube-api-access-d8p9h" pod "a76fafcf-a716-4a3b-86fb-a12a17f1ae5e" (UID: "a76fafcf-a716-4a3b-86fb-a12a17f1ae5e"). InnerVolumeSpecName "kube-api-access-d8p9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:34:50.664771 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.664736 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w85cp\" (UniqueName: \"kubernetes.io/projected/650423de-a897-4e26-b2a8-c822c8642895-kube-api-access-w85cp\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:50.664771 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.664770 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8p9h\" (UniqueName: \"kubernetes.io/projected/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-kube-api-access-d8p9h\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:50.664982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.664783 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:50.664982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:50.664793 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/650423de-a897-4e26-b2a8-c822c8642895-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:51.186973 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.184838 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:51.319642 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.319609 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4" Oct 30 02:34:51.320183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.319608 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4" event={"ID":"a76fafcf-a716-4a3b-86fb-a12a17f1ae5e","Type":"ContainerDied","Data":"f44397b7da65b5abfb85fae0d36c2e2c326ef6364d0cf82538f18d6c53e823f3"} Oct 30 02:34:51.320183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.319800 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f44397b7da65b5abfb85fae0d36c2e2c326ef6364d0cf82538f18d6c53e823f3" Oct 30 02:34:51.321404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.321375 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6" event={"ID":"650423de-a897-4e26-b2a8-c822c8642895","Type":"ContainerDied","Data":"59bc17172e1fb02e2cf6343b73f7aaf9c53d563e469840dd70373f78943ecfae"} Oct 30 02:34:51.321404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.321391 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6" Oct 30 02:34:51.321610 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.321420 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59bc17172e1fb02e2cf6343b73f7aaf9c53d563e469840dd70373f78943ecfae" Oct 30 02:34:51.761172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.761136 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:34:51.881029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.880993 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-credential-socket\") pod \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " Oct 30 02:34:51.881233 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.881082 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-envoy\") pod \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " Oct 30 02:34:51.881233 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.881150 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-kube-api-access-rxdkm\" (UniqueName: \"kubernetes.io/projected/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-kube-api-access-rxdkm\") pod \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " Oct 30 02:34:51.881233 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.881188 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-token\" (UniqueName: \"kubernetes.io/projected/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-token\") pod \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " Oct 30 02:34:51.881418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.881264 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istiod-ca-cert\") pod \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " Oct 30 02:34:51.881418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.881320 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-data\") pod \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " Oct 30 02:34:51.881418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.881360 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-workload-socket\") pod \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " Oct 30 02:34:51.881418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.881383 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-credential-socket" pod "3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" (UID: "3d1e7fb2-51e7-442a-ae7f-e19d9671c65b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:34:51.881825 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.881490 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-workload-certs\") pod \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " Oct 30 02:34:51.881825 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.881580 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-podinfo\") pod \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\" (UID: \"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b\") " Oct 30 02:34:51.881825 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.881594 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-workload-socket" pod "3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" (UID: "3d1e7fb2-51e7-442a-ae7f-e19d9671c65b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:34:51.881825 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.881783 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-data" pod "3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" (UID: "3d1e7fb2-51e7-442a-ae7f-e19d9671c65b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:34:51.882053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.881848 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-workload-certs" pod "3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" (UID: "3d1e7fb2-51e7-442a-ae7f-e19d9671c65b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:34:51.882053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.881926 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istiod-ca-cert" pod "3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" (UID: "3d1e7fb2-51e7-442a-ae7f-e19d9671c65b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:34:51.882484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.882461 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:51.882578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.882487 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:51.882578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.882502 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:51.882578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.882513 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:51.882578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.882525 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:51.883820 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.883785 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-kube-api-access-rxdkm" pod "3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" (UID: "3d1e7fb2-51e7-442a-ae7f-e19d9671c65b"). InnerVolumeSpecName "kube-api-access-rxdkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:34:51.884012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.883981 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-token" pod "3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" (UID: "3d1e7fb2-51e7-442a-ae7f-e19d9671c65b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:34:51.884173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.884034 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-envoy" pod "3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" (UID: "3d1e7fb2-51e7-442a-ae7f-e19d9671c65b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:34:51.884173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.884119 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-podinfo" pod "3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" (UID: "3d1e7fb2-51e7-442a-ae7f-e19d9671c65b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:34:51.984034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.983995 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:51.984034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.984031 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxdkm\" (UniqueName: \"kubernetes.io/projected/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-kube-api-access-rxdkm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:51.984034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.984043 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:51.984324 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:51.984052 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:34:52.084940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:52.084839 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:52.208474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:52.208435 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:52.333348 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:52.333308 5998 generic.go:356] "Generic (PLEG): container finished" podID="3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" containerID="6b484ea0b153eeb2afe624c009a9e4151c5563e55616c9a89173992d840e1a45" exitCode=0 Oct 30 02:34:52.333855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:52.333378 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" Oct 30 02:34:52.333855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:52.333413 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" event={"ID":"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b","Type":"ContainerDied","Data":"6b484ea0b153eeb2afe624c009a9e4151c5563e55616c9a89173992d840e1a45"} Oct 30 02:34:52.333855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:52.333461 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj" event={"ID":"3d1e7fb2-51e7-442a-ae7f-e19d9671c65b","Type":"ContainerDied","Data":"9c18230d5a068797c98d44cfe7c74e6b1c126fee4ace7d387ee715a283d7df4b"} Oct 30 02:34:52.333855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:52.333494 5998 scope.go:122] "RemoveContainer" containerID="6b484ea0b153eeb2afe624c009a9e4151c5563e55616c9a89173992d840e1a45" Oct 30 02:34:52.347632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:52.347607 5998 scope.go:122] "RemoveContainer" containerID="2d7ea69ae41a5967fd68e4dc2bc2f12c95f6a766bd7de576d29f7b03492bbe5e" Oct 30 02:34:52.360241 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:52.360216 5998 scope.go:122] "RemoveContainer" containerID="6b484ea0b153eeb2afe624c009a9e4151c5563e55616c9a89173992d840e1a45" Oct 30 02:34:52.360581 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:34:52.360549 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b484ea0b153eeb2afe624c009a9e4151c5563e55616c9a89173992d840e1a45\": container with ID starting with 6b484ea0b153eeb2afe624c009a9e4151c5563e55616c9a89173992d840e1a45 not found: ID does not exist" containerID="6b484ea0b153eeb2afe624c009a9e4151c5563e55616c9a89173992d840e1a45" Oct 30 02:34:52.360691 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:52.360586 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b484ea0b153eeb2afe624c009a9e4151c5563e55616c9a89173992d840e1a45"} err="failed to get container status \"6b484ea0b153eeb2afe624c009a9e4151c5563e55616c9a89173992d840e1a45\": rpc error: code = NotFound desc = could not find container \"6b484ea0b153eeb2afe624c009a9e4151c5563e55616c9a89173992d840e1a45\": container with ID starting with 6b484ea0b153eeb2afe624c009a9e4151c5563e55616c9a89173992d840e1a45 not found: ID does not exist" Oct 30 02:34:52.360691 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:52.360612 5998 scope.go:122] "RemoveContainer" containerID="2d7ea69ae41a5967fd68e4dc2bc2f12c95f6a766bd7de576d29f7b03492bbe5e" Oct 30 02:34:52.360891 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:34:52.360868 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d7ea69ae41a5967fd68e4dc2bc2f12c95f6a766bd7de576d29f7b03492bbe5e\": container with ID starting with 2d7ea69ae41a5967fd68e4dc2bc2f12c95f6a766bd7de576d29f7b03492bbe5e not found: ID does not exist" containerID="2d7ea69ae41a5967fd68e4dc2bc2f12c95f6a766bd7de576d29f7b03492bbe5e" Oct 30 02:34:52.360947 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:52.360898 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d7ea69ae41a5967fd68e4dc2bc2f12c95f6a766bd7de576d29f7b03492bbe5e"} err="failed to get container status \"2d7ea69ae41a5967fd68e4dc2bc2f12c95f6a766bd7de576d29f7b03492bbe5e\": rpc error: code = NotFound desc = could not find container \"2d7ea69ae41a5967fd68e4dc2bc2f12c95f6a766bd7de576d29f7b03492bbe5e\": container with ID starting with 2d7ea69ae41a5967fd68e4dc2bc2f12c95f6a766bd7de576d29f7b03492bbe5e not found: ID does not exist" Oct 30 02:34:52.382485 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:52.382454 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj"] Oct 30 02:34:52.393365 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:52.393335 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d97e4c-openshift-default-7d7b8c58c7-mwggj"] Oct 30 02:34:52.489080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:52.489042 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:52.543920 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:52.543882 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" path="/var/lib/kubelet/pods/3d1e7fb2-51e7-442a-ae7f-e19d9671c65b/volumes" Oct 30 02:34:52.984978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:52.984943 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:54.552311 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:54.552269 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:55.221294 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:34:55.221253 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:34:56.209603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:56.209546 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:57.132114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.131762 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf"] Oct 30 02:34:57.161065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.160990 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:57.775969 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.775926 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j"] Oct 30 02:34:57.776861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.776835 5998 cpu_manager.go:414] "containerMap: removing container" podUID="bf65846d-6629-424c-9eda-4c359fd1992a" containerName="istio-proxy" Oct 30 02:34:57.776861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.776860 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf65846d-6629-424c-9eda-4c359fd1992a" containerName="istio-proxy" Oct 30 02:34:57.777011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.776902 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a76fafcf-a716-4a3b-86fb-a12a17f1ae5e" containerName="revoke-keys" Oct 30 02:34:57.777011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.776910 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a76fafcf-a716-4a3b-86fb-a12a17f1ae5e" containerName="revoke-keys" Oct 30 02:34:57.777011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.776933 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" containerName="istio-proxy" Oct 30 02:34:57.777011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.776939 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" containerName="istio-proxy" Oct 30 02:34:57.777011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.776947 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" containerName="istio-proxy" Oct 30 02:34:57.777011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.776951 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" containerName="istio-proxy" Oct 30 02:34:57.777011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.776962 5998 cpu_manager.go:414] "containerMap: removing container" podUID="650423de-a897-4e26-b2a8-c822c8642895" containerName="revoke-keys" Oct 30 02:34:57.777011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.776967 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="650423de-a897-4e26-b2a8-c822c8642895" containerName="revoke-keys" Oct 30 02:34:57.777437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.777064 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" containerName="istio-proxy" Oct 30 02:34:57.777437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.777077 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1e7fb2-51e7-442a-ae7f-e19d9671c65b" containerName="istio-proxy" Oct 30 02:34:57.777437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.777084 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="650423de-a897-4e26-b2a8-c822c8642895" containerName="revoke-keys" Oct 30 02:34:57.777437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.777105 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf65846d-6629-424c-9eda-4c359fd1992a" containerName="istio-proxy" Oct 30 02:34:57.777437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.777117 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf65846d-6629-424c-9eda-4c359fd1992a" containerName="istio-proxy" Oct 30 02:34:57.777437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.777133 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a76fafcf-a716-4a3b-86fb-a12a17f1ae5e" containerName="revoke-keys" Oct 30 02:34:57.782576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.782552 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j" Oct 30 02:34:57.801336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.801309 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:34:57.811008 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.810972 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:34:57.811008 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.810995 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:34:57.811243 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.811054 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:34:57.845158 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.832388 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j"] Oct 30 02:34:57.906114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.899817 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:57.934114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.933719 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48n2m\" (UniqueName: \"kubernetes.io/projected/1fd3f37a-562d-4b69-b061-7c26891d0c43-kube-api-access-48n2m\") pod \"maas-api-revoke-sub-987cba5c-zl58j\" (UID: \"1fd3f37a-562d-4b69-b061-7c26891d0c43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j" Oct 30 02:34:57.934114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.933969 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd3f37a-562d-4b69-b061-7c26891d0c43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-987cba5c-zl58j\" (UID: \"1fd3f37a-562d-4b69-b061-7c26891d0c43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j" Oct 30 02:34:57.938115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:57.934451 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd3f37a-562d-4b69-b061-7c26891d0c43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-987cba5c-zl58j\" (UID: \"1fd3f37a-562d-4b69-b061-7c26891d0c43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j" Oct 30 02:34:58.038923 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.038612 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd3f37a-562d-4b69-b061-7c26891d0c43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-987cba5c-zl58j\" (UID: \"1fd3f37a-562d-4b69-b061-7c26891d0c43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j" Oct 30 02:34:58.039480 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.039450 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd3f37a-562d-4b69-b061-7c26891d0c43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-987cba5c-zl58j\" (UID: \"1fd3f37a-562d-4b69-b061-7c26891d0c43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j" Oct 30 02:34:58.039746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.039716 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48n2m\" (UniqueName: \"kubernetes.io/projected/1fd3f37a-562d-4b69-b061-7c26891d0c43-kube-api-access-48n2m\") pod \"maas-api-revoke-sub-987cba5c-zl58j\" (UID: \"1fd3f37a-562d-4b69-b061-7c26891d0c43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j" Oct 30 02:34:58.040373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.040345 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd3f37a-562d-4b69-b061-7c26891d0c43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-987cba5c-zl58j\" (UID: \"1fd3f37a-562d-4b69-b061-7c26891d0c43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j" Oct 30 02:34:58.042878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.042805 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd3f37a-562d-4b69-b061-7c26891d0c43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-987cba5c-zl58j\" (UID: \"1fd3f37a-562d-4b69-b061-7c26891d0c43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j" Oct 30 02:34:58.066707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.066674 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48n2m\" (UniqueName: \"kubernetes.io/projected/1fd3f37a-562d-4b69-b061-7c26891d0c43-kube-api-access-48n2m\") pod \"maas-api-revoke-sub-987cba5c-zl58j\" (UID: \"1fd3f37a-562d-4b69-b061-7c26891d0c43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j" Oct 30 02:34:58.097576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.097537 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j" Oct 30 02:34:58.109506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.109471 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:34:58.264112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.263115 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5"] Oct 30 02:34:58.264304 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.264175 5998 cpu_manager.go:414] "containerMap: removing container" podUID="bf65846d-6629-424c-9eda-4c359fd1992a" containerName="istio-proxy" Oct 30 02:34:58.264304 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.264198 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf65846d-6629-424c-9eda-4c359fd1992a" containerName="istio-proxy" Oct 30 02:34:58.270293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.270261 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5" Oct 30 02:34:58.273429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.273331 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5"] Oct 30 02:34:58.427003 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.426979 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j"] Oct 30 02:34:58.430168 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:34:58.430134 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd3f37a_562d_4b69_b061_7c26891d0c43.slice/crio-e74f3601e4f9edbee5f91ce042ffde6512d85197d58db9f153b6723d9f1c7cb8 WatchSource:0}: Error finding container e74f3601e4f9edbee5f91ce042ffde6512d85197d58db9f153b6723d9f1c7cb8: Status 404 returned error can't find the container with id e74f3601e4f9edbee5f91ce042ffde6512d85197d58db9f153b6723d9f1c7cb8 Oct 30 02:34:58.457055 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.457021 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10a5bfc-b78e-4075-b194-71c14f8e0a80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65479906-qqbt5\" (UID: \"a10a5bfc-b78e-4075-b194-71c14f8e0a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5" Oct 30 02:34:58.457299 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.457277 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10a5bfc-b78e-4075-b194-71c14f8e0a80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65479906-qqbt5\" (UID: \"a10a5bfc-b78e-4075-b194-71c14f8e0a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5" Oct 30 02:34:58.457386 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.457323 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2p5x\" (UniqueName: \"kubernetes.io/projected/a10a5bfc-b78e-4075-b194-71c14f8e0a80-kube-api-access-d2p5x\") pod \"maas-api-revoke-sub-65479906-qqbt5\" (UID: \"a10a5bfc-b78e-4075-b194-71c14f8e0a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5" Oct 30 02:34:58.558714 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.558669 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10a5bfc-b78e-4075-b194-71c14f8e0a80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65479906-qqbt5\" (UID: \"a10a5bfc-b78e-4075-b194-71c14f8e0a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5" Oct 30 02:34:58.558919 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.558722 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2p5x\" (UniqueName: \"kubernetes.io/projected/a10a5bfc-b78e-4075-b194-71c14f8e0a80-kube-api-access-d2p5x\") pod \"maas-api-revoke-sub-65479906-qqbt5\" (UID: \"a10a5bfc-b78e-4075-b194-71c14f8e0a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5" Oct 30 02:34:58.559004 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.558921 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10a5bfc-b78e-4075-b194-71c14f8e0a80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65479906-qqbt5\" (UID: \"a10a5bfc-b78e-4075-b194-71c14f8e0a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5" Oct 30 02:34:58.559769 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.559743 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10a5bfc-b78e-4075-b194-71c14f8e0a80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65479906-qqbt5\" (UID: \"a10a5bfc-b78e-4075-b194-71c14f8e0a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5" Oct 30 02:34:58.561235 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.561217 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10a5bfc-b78e-4075-b194-71c14f8e0a80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65479906-qqbt5\" (UID: \"a10a5bfc-b78e-4075-b194-71c14f8e0a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5" Oct 30 02:34:58.580656 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.580598 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2p5x\" (UniqueName: \"kubernetes.io/projected/a10a5bfc-b78e-4075-b194-71c14f8e0a80-kube-api-access-d2p5x\") pod \"maas-api-revoke-sub-65479906-qqbt5\" (UID: \"a10a5bfc-b78e-4075-b194-71c14f8e0a80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5" Oct 30 02:34:58.620033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.619992 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5" Oct 30 02:34:58.952280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:58.952250 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5"] Oct 30 02:34:58.953727 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:34:58.953693 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10a5bfc_b78e_4075_b194_71c14f8e0a80.slice/crio-eba717a6e6f6e0b84f37ec17b28c3245154145a06d5425561f521ed2561bff90 WatchSource:0}: Error finding container eba717a6e6f6e0b84f37ec17b28c3245154145a06d5425561f521ed2561bff90: Status 404 returned error can't find the container with id eba717a6e6f6e0b84f37ec17b28c3245154145a06d5425561f521ed2561bff90 Oct 30 02:34:59.429416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:59.429379 5998 generic.go:356] "Generic (PLEG): container finished" podID="1fd3f37a-562d-4b69-b061-7c26891d0c43" containerID="5f9dc054e7db368be5c33f46b7b2ee4502b5b3eabe309d2103213fcf16e99415" exitCode=0 Oct 30 02:34:59.429612 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:59.429463 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j" event={"ID":"1fd3f37a-562d-4b69-b061-7c26891d0c43","Type":"ContainerDied","Data":"5f9dc054e7db368be5c33f46b7b2ee4502b5b3eabe309d2103213fcf16e99415"} Oct 30 02:34:59.429612 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:59.429516 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j" event={"ID":"1fd3f37a-562d-4b69-b061-7c26891d0c43","Type":"ContainerStarted","Data":"e74f3601e4f9edbee5f91ce042ffde6512d85197d58db9f153b6723d9f1c7cb8"} Oct 30 02:34:59.431501 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:59.431464 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5" event={"ID":"a10a5bfc-b78e-4075-b194-71c14f8e0a80","Type":"ContainerStarted","Data":"5036786471da0ccef4efbd4f2ad228c4b137d01fdec12980f404e9d4ea036cff"} Oct 30 02:34:59.431654 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:59.431524 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5" event={"ID":"a10a5bfc-b78e-4075-b194-71c14f8e0a80","Type":"ContainerStarted","Data":"eba717a6e6f6e0b84f37ec17b28c3245154145a06d5425561f521ed2561bff90"} Oct 30 02:34:59.507856 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:59.507743 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5" podStartSLOduration=1.5077262280000001 podStartE2EDuration="1.507726228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:34:59.503815297 +0000 UTC m=+3929.082606380" watchObservedRunningTime="2026-10-30 02:34:59.507726228 +0000 UTC m=+3929.086517354" Oct 30 02:34:59.764507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:34:59.764402 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:00.444019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:00.443981 5998 generic.go:356] "Generic (PLEG): container finished" podID="a10a5bfc-b78e-4075-b194-71c14f8e0a80" containerID="5036786471da0ccef4efbd4f2ad228c4b137d01fdec12980f404e9d4ea036cff" exitCode=0 Oct 30 02:35:00.444624 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:00.444066 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5" event={"ID":"a10a5bfc-b78e-4075-b194-71c14f8e0a80","Type":"ContainerDied","Data":"5036786471da0ccef4efbd4f2ad228c4b137d01fdec12980f404e9d4ea036cff"} Oct 30 02:35:00.528814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:00.528773 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:00.648291 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:00.648261 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j" Oct 30 02:35:00.701822 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:00.701735 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd3f37a-562d-4b69-b061-7c26891d0c43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd3f37a-562d-4b69-b061-7c26891d0c43-maas-api-xks-ca\") pod \"1fd3f37a-562d-4b69-b061-7c26891d0c43\" (UID: \"1fd3f37a-562d-4b69-b061-7c26891d0c43\") " Oct 30 02:35:00.702001 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:00.701909 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd3f37a-562d-4b69-b061-7c26891d0c43-kube-api-access-48n2m\" (UniqueName: \"kubernetes.io/projected/1fd3f37a-562d-4b69-b061-7c26891d0c43-kube-api-access-48n2m\") pod \"1fd3f37a-562d-4b69-b061-7c26891d0c43\" (UID: \"1fd3f37a-562d-4b69-b061-7c26891d0c43\") " Oct 30 02:35:00.702124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:00.702000 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fd3f37a-562d-4b69-b061-7c26891d0c43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd3f37a-562d-4b69-b061-7c26891d0c43-maas-api-openshift-ca\") pod \"1fd3f37a-562d-4b69-b061-7c26891d0c43\" (UID: \"1fd3f37a-562d-4b69-b061-7c26891d0c43\") " Oct 30 02:35:00.704387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:00.704354 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd3f37a-562d-4b69-b061-7c26891d0c43-maas-api-openshift-ca" pod "1fd3f37a-562d-4b69-b061-7c26891d0c43" (UID: "1fd3f37a-562d-4b69-b061-7c26891d0c43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:35:00.705843 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:00.705797 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd3f37a-562d-4b69-b061-7c26891d0c43-kube-api-access-48n2m" pod "1fd3f37a-562d-4b69-b061-7c26891d0c43" (UID: "1fd3f37a-562d-4b69-b061-7c26891d0c43"). InnerVolumeSpecName "kube-api-access-48n2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:00.705962 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:00.705907 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd3f37a-562d-4b69-b061-7c26891d0c43-maas-api-xks-ca" pod "1fd3f37a-562d-4b69-b061-7c26891d0c43" (UID: "1fd3f37a-562d-4b69-b061-7c26891d0c43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:35:00.804035 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:00.803996 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd3f37a-562d-4b69-b061-7c26891d0c43-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:00.804035 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:00.804032 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48n2m\" (UniqueName: \"kubernetes.io/projected/1fd3f37a-562d-4b69-b061-7c26891d0c43-kube-api-access-48n2m\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:00.804358 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:00.804045 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd3f37a-562d-4b69-b061-7c26891d0c43-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:01.456506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:01.456470 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j" Oct 30 02:35:01.456506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:01.456493 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j" event={"ID":"1fd3f37a-562d-4b69-b061-7c26891d0c43","Type":"ContainerDied","Data":"e74f3601e4f9edbee5f91ce042ffde6512d85197d58db9f153b6723d9f1c7cb8"} Oct 30 02:35:01.457144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:01.456555 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e74f3601e4f9edbee5f91ce042ffde6512d85197d58db9f153b6723d9f1c7cb8" Oct 30 02:35:01.484285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:01.484252 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:01.666998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:01.666971 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5" Oct 30 02:35:01.717158 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:01.717064 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10a5bfc-b78e-4075-b194-71c14f8e0a80-kube-api-access-d2p5x\" (UniqueName: \"kubernetes.io/projected/a10a5bfc-b78e-4075-b194-71c14f8e0a80-kube-api-access-d2p5x\") pod \"a10a5bfc-b78e-4075-b194-71c14f8e0a80\" (UID: \"a10a5bfc-b78e-4075-b194-71c14f8e0a80\") " Oct 30 02:35:01.717340 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:01.717295 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10a5bfc-b78e-4075-b194-71c14f8e0a80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10a5bfc-b78e-4075-b194-71c14f8e0a80-maas-api-openshift-ca\") pod \"a10a5bfc-b78e-4075-b194-71c14f8e0a80\" (UID: \"a10a5bfc-b78e-4075-b194-71c14f8e0a80\") " Oct 30 02:35:01.717409 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:01.717382 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a10a5bfc-b78e-4075-b194-71c14f8e0a80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10a5bfc-b78e-4075-b194-71c14f8e0a80-maas-api-xks-ca\") pod \"a10a5bfc-b78e-4075-b194-71c14f8e0a80\" (UID: \"a10a5bfc-b78e-4075-b194-71c14f8e0a80\") " Oct 30 02:35:01.717885 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:01.717857 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10a5bfc-b78e-4075-b194-71c14f8e0a80-maas-api-openshift-ca" pod "a10a5bfc-b78e-4075-b194-71c14f8e0a80" (UID: "a10a5bfc-b78e-4075-b194-71c14f8e0a80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:35:01.718310 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:01.718286 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a10a5bfc-b78e-4075-b194-71c14f8e0a80-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:01.728632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:01.728593 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a10a5bfc-b78e-4075-b194-71c14f8e0a80-maas-api-xks-ca" pod "a10a5bfc-b78e-4075-b194-71c14f8e0a80" (UID: "a10a5bfc-b78e-4075-b194-71c14f8e0a80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:35:01.735437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:01.735392 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10a5bfc-b78e-4075-b194-71c14f8e0a80-kube-api-access-d2p5x" pod "a10a5bfc-b78e-4075-b194-71c14f8e0a80" (UID: "a10a5bfc-b78e-4075-b194-71c14f8e0a80"). InnerVolumeSpecName "kube-api-access-d2p5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:01.819837 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:01.819792 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2p5x\" (UniqueName: \"kubernetes.io/projected/a10a5bfc-b78e-4075-b194-71c14f8e0a80-kube-api-access-d2p5x\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:01.819837 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:01.819832 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a10a5bfc-b78e-4075-b194-71c14f8e0a80-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:02.466756 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:02.466716 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5" event={"ID":"a10a5bfc-b78e-4075-b194-71c14f8e0a80","Type":"ContainerDied","Data":"eba717a6e6f6e0b84f37ec17b28c3245154145a06d5425561f521ed2561bff90"} Oct 30 02:35:02.466756 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:02.466753 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5" Oct 30 02:35:02.467384 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:02.466812 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eba717a6e6f6e0b84f37ec17b28c3245154145a06d5425561f521ed2561bff90" Oct 30 02:35:03.140529 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:03.140486 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:04.630149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:04.630064 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:06.373084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:06.373025 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:07.874924 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:07.874873 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:09.370579 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:09.370519 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:10.923597 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:10.923560 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:12.165527 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.165464 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" podUID="c3524261-ac9b-419e-82b9-c46404256f89" containerName="main" containerID="cri-o://f6168882745d2128ba538e8bb0788437761afbdfa8351e3e3512edfd71044d0b" gracePeriod=45 Oct 30 02:35:12.331338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.331286 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" podUID="c3524261-ac9b-419e-82b9-c46404256f89" containerName="main" probeResult="failure" output="Get \"https://10.133.1.1:8000/ready\": dial tcp 10.133.1.1:8000: connect: connection refused" Oct 30 02:35:12.465496 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.465466 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:35:12.571946 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.571910 5998 generic.go:356] "Generic (PLEG): container finished" podID="c3524261-ac9b-419e-82b9-c46404256f89" containerID="f6168882745d2128ba538e8bb0788437761afbdfa8351e3e3512edfd71044d0b" exitCode=0 Oct 30 02:35:12.572200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.571972 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" Oct 30 02:35:12.572200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.571998 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" event={"ID":"c3524261-ac9b-419e-82b9-c46404256f89","Type":"ContainerDied","Data":"f6168882745d2128ba538e8bb0788437761afbdfa8351e3e3512edfd71044d0b"} Oct 30 02:35:12.572200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.572044 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf" event={"ID":"c3524261-ac9b-419e-82b9-c46404256f89","Type":"ContainerDied","Data":"8a7901b9bcffef25a9127d72a6cc3227f44cccb7830ea09cef17a20fdc406ce5"} Oct 30 02:35:12.572200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.572065 5998 scope.go:122] "RemoveContainer" containerID="f6168882745d2128ba538e8bb0788437761afbdfa8351e3e3512edfd71044d0b" Oct 30 02:35:12.583297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.583277 5998 scope.go:122] "RemoveContainer" containerID="f6168882745d2128ba538e8bb0788437761afbdfa8351e3e3512edfd71044d0b" Oct 30 02:35:12.583647 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:35:12.583615 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6168882745d2128ba538e8bb0788437761afbdfa8351e3e3512edfd71044d0b\": container with ID starting with f6168882745d2128ba538e8bb0788437761afbdfa8351e3e3512edfd71044d0b not found: ID does not exist" containerID="f6168882745d2128ba538e8bb0788437761afbdfa8351e3e3512edfd71044d0b" Oct 30 02:35:12.583758 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.583659 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6168882745d2128ba538e8bb0788437761afbdfa8351e3e3512edfd71044d0b"} err="failed to get container status \"f6168882745d2128ba538e8bb0788437761afbdfa8351e3e3512edfd71044d0b\": rpc error: code = NotFound desc = could not find container \"f6168882745d2128ba538e8bb0788437761afbdfa8351e3e3512edfd71044d0b\": container with ID starting with f6168882745d2128ba538e8bb0788437761afbdfa8351e3e3512edfd71044d0b not found: ID does not exist" Oct 30 02:35:12.613340 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.613302 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:12.634586 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.634555 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-home\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-home\") pod \"c3524261-ac9b-419e-82b9-c46404256f89\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " Oct 30 02:35:12.634873 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.634848 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-tmp-dir\") pod \"c3524261-ac9b-419e-82b9-c46404256f89\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " Oct 30 02:35:12.634941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.634885 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-model-cache\") pod \"c3524261-ac9b-419e-82b9-c46404256f89\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " Oct 30 02:35:12.634941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.634902 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-home" pod "c3524261-ac9b-419e-82b9-c46404256f89" (UID: "c3524261-ac9b-419e-82b9-c46404256f89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:12.634941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.634925 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3524261-ac9b-419e-82b9-c46404256f89-kube-api-access-zr6dx\" (UniqueName: \"kubernetes.io/projected/c3524261-ac9b-419e-82b9-c46404256f89-kube-api-access-zr6dx\") pod \"c3524261-ac9b-419e-82b9-c46404256f89\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " Oct 30 02:35:12.635136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.634983 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3524261-ac9b-419e-82b9-c46404256f89-tls-certs\" (UniqueName: \"kubernetes.io/secret/c3524261-ac9b-419e-82b9-c46404256f89-tls-certs\") pod \"c3524261-ac9b-419e-82b9-c46404256f89\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " Oct 30 02:35:12.635136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.635054 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-dshm\") pod \"c3524261-ac9b-419e-82b9-c46404256f89\" (UID: \"c3524261-ac9b-419e-82b9-c46404256f89\") " Oct 30 02:35:12.635252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.635187 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-tmp-dir" pod "c3524261-ac9b-419e-82b9-c46404256f89" (UID: "c3524261-ac9b-419e-82b9-c46404256f89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:12.635252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.635211 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-model-cache" pod "c3524261-ac9b-419e-82b9-c46404256f89" (UID: "c3524261-ac9b-419e-82b9-c46404256f89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:12.635790 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.635766 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:12.635915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.635793 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:12.635915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.635808 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:12.637762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.637674 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3524261-ac9b-419e-82b9-c46404256f89-tls-certs" pod "c3524261-ac9b-419e-82b9-c46404256f89" (UID: "c3524261-ac9b-419e-82b9-c46404256f89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:35:12.637905 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.637760 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3524261-ac9b-419e-82b9-c46404256f89-kube-api-access-zr6dx" pod "c3524261-ac9b-419e-82b9-c46404256f89" (UID: "c3524261-ac9b-419e-82b9-c46404256f89"). InnerVolumeSpecName "kube-api-access-zr6dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:12.638156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.638135 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-dshm" pod "c3524261-ac9b-419e-82b9-c46404256f89" (UID: "c3524261-ac9b-419e-82b9-c46404256f89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:12.736893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.736793 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr6dx\" (UniqueName: \"kubernetes.io/projected/c3524261-ac9b-419e-82b9-c46404256f89-kube-api-access-zr6dx\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:12.736893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.736834 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3524261-ac9b-419e-82b9-c46404256f89-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:12.736893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.736850 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3524261-ac9b-419e-82b9-c46404256f89-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:12.963201 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.963154 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf"] Oct 30 02:35:12.972857 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:12.969825 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-db01c4/test-model-db01c4-kserve-786876d559-w8cbf"] Oct 30 02:35:14.540994 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:14.540960 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3524261-ac9b-419e-82b9-c46404256f89" path="/var/lib/kubelet/pods/c3524261-ac9b-419e-82b9-c46404256f89/volumes" Oct 30 02:35:14.563689 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:14.563660 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:16.506202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:16.506160 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:18.277424 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:18.277355 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t"] Oct 30 02:35:18.336635 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:18.336596 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:18.463114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:18.460586 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:18.732821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:18.732775 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:19.518225 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:19.518190 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:19.916375 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:19.916334 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:20.277339 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:20.277259 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:21.293791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:21.293747 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87"] Oct 30 02:35:21.294331 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:21.293977 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" podUID="c13ee6d9-ce09-49b9-830f-5b8b25506581" containerName="istio-proxy" containerID="cri-o://4883db70d3ca9e4126e230d48adbf4f534998dc62c149f7f27bc8be7c0d4ef6b" gracePeriod=30 Oct 30 02:35:21.858650 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:21.858612 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:23.094918 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:23.094883 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x"] Oct 30 02:35:23.095363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:23.095084 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" podUID="d1b71a2d-53c4-4e41-85ff-b82b82292598" containerName="istio-proxy" containerID="cri-o://958467014e0618b5aeab19dc44c3175db9f4679fd947bc3af0f2b569ee3c2842" gracePeriod=30 Oct 30 02:35:23.488890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:23.488849 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:24.391605 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:24.391563 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:24.525861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:24.525772 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:24.881486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:24.881415 5998 patch_prober.go:29] interesting pod/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:35:24.881731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:24.881501 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" podUID="c13ee6d9-ce09-49b9-830f-5b8b25506581" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:35:25.153497 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:25.153394 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:25.225203 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:35:25.225158 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:35:26.583815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.583789 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:35:26.690551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.690486 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-token\" (UniqueName: \"kubernetes.io/projected/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-token\") pod \"c13ee6d9-ce09-49b9-830f-5b8b25506581\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " Oct 30 02:35:26.690803 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.690720 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c13ee6d9-ce09-49b9-830f-5b8b25506581-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13ee6d9-ce09-49b9-830f-5b8b25506581-istiod-ca-cert\") pod \"c13ee6d9-ce09-49b9-830f-5b8b25506581\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " Oct 30 02:35:26.690910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.690852 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-envoy\") pod \"c13ee6d9-ce09-49b9-830f-5b8b25506581\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " Oct 30 02:35:26.690910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.690897 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-workload-socket\") pod \"c13ee6d9-ce09-49b9-830f-5b8b25506581\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " Oct 30 02:35:26.691043 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.690927 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-credential-socket\") pod \"c13ee6d9-ce09-49b9-830f-5b8b25506581\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " Oct 30 02:35:26.691043 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.690958 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13ee6d9-ce09-49b9-830f-5b8b25506581-kube-api-access-5rjwb\" (UniqueName: \"kubernetes.io/projected/c13ee6d9-ce09-49b9-830f-5b8b25506581-kube-api-access-5rjwb\") pod \"c13ee6d9-ce09-49b9-830f-5b8b25506581\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " Oct 30 02:35:26.691043 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.691021 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-podinfo\") pod \"c13ee6d9-ce09-49b9-830f-5b8b25506581\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " Oct 30 02:35:26.691260 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.691047 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-data\") pod \"c13ee6d9-ce09-49b9-830f-5b8b25506581\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " Oct 30 02:35:26.691260 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.691147 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-workload-certs\") pod \"c13ee6d9-ce09-49b9-830f-5b8b25506581\" (UID: \"c13ee6d9-ce09-49b9-830f-5b8b25506581\") " Oct 30 02:35:26.691384 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.691324 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-workload-socket" pod "c13ee6d9-ce09-49b9-830f-5b8b25506581" (UID: "c13ee6d9-ce09-49b9-830f-5b8b25506581"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:26.691384 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.691329 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13ee6d9-ce09-49b9-830f-5b8b25506581-istiod-ca-cert" pod "c13ee6d9-ce09-49b9-830f-5b8b25506581" (UID: "c13ee6d9-ce09-49b9-830f-5b8b25506581"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:35:26.691563 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.691541 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-data" pod "c13ee6d9-ce09-49b9-830f-5b8b25506581" (UID: "c13ee6d9-ce09-49b9-830f-5b8b25506581"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:26.691800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.691768 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-workload-certs" pod "c13ee6d9-ce09-49b9-830f-5b8b25506581" (UID: "c13ee6d9-ce09-49b9-830f-5b8b25506581"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:26.691900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.691851 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c13ee6d9-ce09-49b9-830f-5b8b25506581-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:26.691900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.691875 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:26.691900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.691889 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:26.692167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.692107 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-credential-socket" pod "c13ee6d9-ce09-49b9-830f-5b8b25506581" (UID: "c13ee6d9-ce09-49b9-830f-5b8b25506581"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:26.693431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.693362 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-token" pod "c13ee6d9-ce09-49b9-830f-5b8b25506581" (UID: "c13ee6d9-ce09-49b9-830f-5b8b25506581"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:26.694451 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.694426 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13ee6d9-ce09-49b9-830f-5b8b25506581-kube-api-access-5rjwb" pod "c13ee6d9-ce09-49b9-830f-5b8b25506581" (UID: "c13ee6d9-ce09-49b9-830f-5b8b25506581"). InnerVolumeSpecName "kube-api-access-5rjwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:26.694559 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.694461 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-podinfo" pod "c13ee6d9-ce09-49b9-830f-5b8b25506581" (UID: "c13ee6d9-ce09-49b9-830f-5b8b25506581"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:35:26.694718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.694685 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-envoy" pod "c13ee6d9-ce09-49b9-830f-5b8b25506581" (UID: "c13ee6d9-ce09-49b9-830f-5b8b25506581"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:26.769450 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.769413 5998 generic.go:356] "Generic (PLEG): container finished" podID="c13ee6d9-ce09-49b9-830f-5b8b25506581" containerID="4883db70d3ca9e4126e230d48adbf4f534998dc62c149f7f27bc8be7c0d4ef6b" exitCode=0 Oct 30 02:35:26.769628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.769479 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" Oct 30 02:35:26.769628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.769506 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" event={"ID":"c13ee6d9-ce09-49b9-830f-5b8b25506581","Type":"ContainerDied","Data":"4883db70d3ca9e4126e230d48adbf4f534998dc62c149f7f27bc8be7c0d4ef6b"} Oct 30 02:35:26.769628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.769608 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87" event={"ID":"c13ee6d9-ce09-49b9-830f-5b8b25506581","Type":"ContainerDied","Data":"dcf44a2e0356db513113193fccc341a72e009c878c2ca9f4400bf163c35ea11c"} Oct 30 02:35:26.769771 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.769641 5998 scope.go:122] "RemoveContainer" containerID="4883db70d3ca9e4126e230d48adbf4f534998dc62c149f7f27bc8be7c0d4ef6b" Oct 30 02:35:26.773346 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.773291 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:26.782159 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.782140 5998 scope.go:122] "RemoveContainer" containerID="4883db70d3ca9e4126e230d48adbf4f534998dc62c149f7f27bc8be7c0d4ef6b" Oct 30 02:35:26.782512 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:35:26.782486 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4883db70d3ca9e4126e230d48adbf4f534998dc62c149f7f27bc8be7c0d4ef6b\": container with ID starting with 4883db70d3ca9e4126e230d48adbf4f534998dc62c149f7f27bc8be7c0d4ef6b not found: ID does not exist" containerID="4883db70d3ca9e4126e230d48adbf4f534998dc62c149f7f27bc8be7c0d4ef6b" Oct 30 02:35:26.782603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.782527 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4883db70d3ca9e4126e230d48adbf4f534998dc62c149f7f27bc8be7c0d4ef6b"} err="failed to get container status \"4883db70d3ca9e4126e230d48adbf4f534998dc62c149f7f27bc8be7c0d4ef6b\": rpc error: code = NotFound desc = could not find container \"4883db70d3ca9e4126e230d48adbf4f534998dc62c149f7f27bc8be7c0d4ef6b\": container with ID starting with 4883db70d3ca9e4126e230d48adbf4f534998dc62c149f7f27bc8be7c0d4ef6b not found: ID does not exist" Oct 30 02:35:26.793166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.793138 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:26.793281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.793171 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:26.793281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.793187 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rjwb\" (UniqueName: \"kubernetes.io/projected/c13ee6d9-ce09-49b9-830f-5b8b25506581-kube-api-access-5rjwb\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:26.793281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.793202 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:26.793281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.793215 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c13ee6d9-ce09-49b9-830f-5b8b25506581-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:26.793281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.793228 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c13ee6d9-ce09-49b9-830f-5b8b25506581-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:26.858679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.850605 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87"] Oct 30 02:35:26.864133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:26.863003 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b93d51-openshift-default-7f57fdf98f-mcc87"] Oct 30 02:35:27.089595 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:27.089564 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:27.260497 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:27.260453 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:27.383337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:27.383287 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:27.835141 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:27.828432 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:28.390737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.390703 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:35:28.452510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.452473 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-workload-certs\") pod \"d1b71a2d-53c4-4e41-85ff-b82b82292598\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " Oct 30 02:35:28.452699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.452535 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b71a2d-53c4-4e41-85ff-b82b82292598-kube-api-access-kkmzt\" (UniqueName: \"kubernetes.io/projected/d1b71a2d-53c4-4e41-85ff-b82b82292598-kube-api-access-kkmzt\") pod \"d1b71a2d-53c4-4e41-85ff-b82b82292598\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " Oct 30 02:35:28.452699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.452592 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-token\" (UniqueName: \"kubernetes.io/projected/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-token\") pod \"d1b71a2d-53c4-4e41-85ff-b82b82292598\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " Oct 30 02:35:28.452826 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.452712 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-data\") pod \"d1b71a2d-53c4-4e41-85ff-b82b82292598\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " Oct 30 02:35:28.452826 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.452758 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1b71a2d-53c4-4e41-85ff-b82b82292598-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b71a2d-53c4-4e41-85ff-b82b82292598-istiod-ca-cert\") pod \"d1b71a2d-53c4-4e41-85ff-b82b82292598\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " Oct 30 02:35:28.452826 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.452789 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-workload-certs" pod "d1b71a2d-53c4-4e41-85ff-b82b82292598" (UID: "d1b71a2d-53c4-4e41-85ff-b82b82292598"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:28.452826 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.452800 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-podinfo\") pod \"d1b71a2d-53c4-4e41-85ff-b82b82292598\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " Oct 30 02:35:28.453079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.452946 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-credential-socket\") pod \"d1b71a2d-53c4-4e41-85ff-b82b82292598\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " Oct 30 02:35:28.453079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.453042 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-data" pod "d1b71a2d-53c4-4e41-85ff-b82b82292598" (UID: "d1b71a2d-53c4-4e41-85ff-b82b82292598"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:28.453258 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.453141 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-envoy\") pod \"d1b71a2d-53c4-4e41-85ff-b82b82292598\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " Oct 30 02:35:28.453258 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.453188 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b71a2d-53c4-4e41-85ff-b82b82292598-istiod-ca-cert" pod "d1b71a2d-53c4-4e41-85ff-b82b82292598" (UID: "d1b71a2d-53c4-4e41-85ff-b82b82292598"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:35:28.453258 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.453228 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-workload-socket\") pod \"d1b71a2d-53c4-4e41-85ff-b82b82292598\" (UID: \"d1b71a2d-53c4-4e41-85ff-b82b82292598\") " Oct 30 02:35:28.453439 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.453407 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-credential-socket" pod "d1b71a2d-53c4-4e41-85ff-b82b82292598" (UID: "d1b71a2d-53c4-4e41-85ff-b82b82292598"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:28.453817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.453789 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-workload-socket" pod "d1b71a2d-53c4-4e41-85ff-b82b82292598" (UID: "d1b71a2d-53c4-4e41-85ff-b82b82292598"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:28.453897 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.453804 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:28.453897 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.453840 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:28.453897 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.453854 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1b71a2d-53c4-4e41-85ff-b82b82292598-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:28.453897 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.453868 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:28.455551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.455524 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-token" pod "d1b71a2d-53c4-4e41-85ff-b82b82292598" (UID: "d1b71a2d-53c4-4e41-85ff-b82b82292598"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:28.456347 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.456319 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-envoy" pod "d1b71a2d-53c4-4e41-85ff-b82b82292598" (UID: "d1b71a2d-53c4-4e41-85ff-b82b82292598"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:28.456473 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.456448 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-podinfo" pod "d1b71a2d-53c4-4e41-85ff-b82b82292598" (UID: "d1b71a2d-53c4-4e41-85ff-b82b82292598"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:35:28.456791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.456769 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b71a2d-53c4-4e41-85ff-b82b82292598-kube-api-access-kkmzt" pod "d1b71a2d-53c4-4e41-85ff-b82b82292598" (UID: "d1b71a2d-53c4-4e41-85ff-b82b82292598"). InnerVolumeSpecName "kube-api-access-kkmzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:28.541936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.541892 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c13ee6d9-ce09-49b9-830f-5b8b25506581" path="/var/lib/kubelet/pods/c13ee6d9-ce09-49b9-830f-5b8b25506581/volumes" Oct 30 02:35:28.559226 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.559186 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:28.559226 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.559225 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1b71a2d-53c4-4e41-85ff-b82b82292598-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:28.559495 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.559238 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkmzt\" (UniqueName: \"kubernetes.io/projected/d1b71a2d-53c4-4e41-85ff-b82b82292598-kube-api-access-kkmzt\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:28.559495 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.559253 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:28.559495 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.559267 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1b71a2d-53c4-4e41-85ff-b82b82292598-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:28.788872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.788836 5998 generic.go:356] "Generic (PLEG): container finished" podID="d1b71a2d-53c4-4e41-85ff-b82b82292598" containerID="958467014e0618b5aeab19dc44c3175db9f4679fd947bc3af0f2b569ee3c2842" exitCode=0 Oct 30 02:35:28.789066 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.788883 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" Oct 30 02:35:28.789066 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.788901 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" event={"ID":"d1b71a2d-53c4-4e41-85ff-b82b82292598","Type":"ContainerDied","Data":"958467014e0618b5aeab19dc44c3175db9f4679fd947bc3af0f2b569ee3c2842"} Oct 30 02:35:28.789066 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.788961 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x" event={"ID":"d1b71a2d-53c4-4e41-85ff-b82b82292598","Type":"ContainerDied","Data":"ff158feacde0ca130b920b2efeb0b40b183928d9129c6be22d316b6c34626a20"} Oct 30 02:35:28.789066 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.788982 5998 scope.go:122] "RemoveContainer" containerID="958467014e0618b5aeab19dc44c3175db9f4679fd947bc3af0f2b569ee3c2842" Oct 30 02:35:28.792723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.792692 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:28.800866 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.800839 5998 scope.go:122] "RemoveContainer" containerID="958467014e0618b5aeab19dc44c3175db9f4679fd947bc3af0f2b569ee3c2842" Oct 30 02:35:28.801217 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:35:28.801186 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"958467014e0618b5aeab19dc44c3175db9f4679fd947bc3af0f2b569ee3c2842\": container with ID starting with 958467014e0618b5aeab19dc44c3175db9f4679fd947bc3af0f2b569ee3c2842 not found: ID does not exist" containerID="958467014e0618b5aeab19dc44c3175db9f4679fd947bc3af0f2b569ee3c2842" Oct 30 02:35:28.801318 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.801229 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"958467014e0618b5aeab19dc44c3175db9f4679fd947bc3af0f2b569ee3c2842"} err="failed to get container status \"958467014e0618b5aeab19dc44c3175db9f4679fd947bc3af0f2b569ee3c2842\": rpc error: code = NotFound desc = could not find container \"958467014e0618b5aeab19dc44c3175db9f4679fd947bc3af0f2b569ee3c2842\": container with ID starting with 958467014e0618b5aeab19dc44c3175db9f4679fd947bc3af0f2b569ee3c2842 not found: ID does not exist" Oct 30 02:35:28.858112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.856954 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x"] Oct 30 02:35:28.874817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:28.874780 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-9dfb3c-openshift-default-89b9d99fc-mpz2x"] Oct 30 02:35:30.355175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:30.355133 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:30.540306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:30.540268 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1b71a2d-53c4-4e41-85ff-b82b82292598" path="/var/lib/kubelet/pods/d1b71a2d-53c4-4e41-85ff-b82b82292598/volumes" Oct 30 02:35:31.892901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:31.892868 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:33.302042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.301997 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" podUID="f2daa657-48ed-40a5-b5ab-3e1596a707ed" containerName="main" containerID="cri-o://f1bc8eb987126f4f8c19eb637e88ccc35af9e2a29ca69c631debc7a2ae7390ac" gracePeriod=45 Oct 30 02:35:33.305071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.304883 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:33.482362 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.482303 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:33.624712 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.624683 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:35:33.675752 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.675701 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-tmp-dir\") pod \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " Oct 30 02:35:33.675961 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.675771 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-dshm\") pod \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " Oct 30 02:35:33.676494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.676457 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-tmp-dir" pod "f2daa657-48ed-40a5-b5ab-3e1596a707ed" (UID: "f2daa657-48ed-40a5-b5ab-3e1596a707ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:33.678414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.678388 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2daa657-48ed-40a5-b5ab-3e1596a707ed-kube-api-access-xrc42\" (UniqueName: \"kubernetes.io/projected/f2daa657-48ed-40a5-b5ab-3e1596a707ed-kube-api-access-xrc42\") pod \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " Oct 30 02:35:33.678540 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.678472 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2daa657-48ed-40a5-b5ab-3e1596a707ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2daa657-48ed-40a5-b5ab-3e1596a707ed-tls-certs\") pod \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " Oct 30 02:35:33.678867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.678519 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-home\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-home\") pod \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " Oct 30 02:35:33.679023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.678995 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-model-cache\") pod \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\" (UID: \"f2daa657-48ed-40a5-b5ab-3e1596a707ed\") " Oct 30 02:35:33.679833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.679619 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:33.679930 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.679865 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-model-cache" pod "f2daa657-48ed-40a5-b5ab-3e1596a707ed" (UID: "f2daa657-48ed-40a5-b5ab-3e1596a707ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:33.680560 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.680535 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-home" pod "f2daa657-48ed-40a5-b5ab-3e1596a707ed" (UID: "f2daa657-48ed-40a5-b5ab-3e1596a707ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:33.681356 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.681320 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2daa657-48ed-40a5-b5ab-3e1596a707ed-tls-certs" pod "f2daa657-48ed-40a5-b5ab-3e1596a707ed" (UID: "f2daa657-48ed-40a5-b5ab-3e1596a707ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:35:33.681449 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.681351 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-dshm" pod "f2daa657-48ed-40a5-b5ab-3e1596a707ed" (UID: "f2daa657-48ed-40a5-b5ab-3e1596a707ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:33.684198 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.684171 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2daa657-48ed-40a5-b5ab-3e1596a707ed-kube-api-access-xrc42" pod "f2daa657-48ed-40a5-b5ab-3e1596a707ed" (UID: "f2daa657-48ed-40a5-b5ab-3e1596a707ed"). InnerVolumeSpecName "kube-api-access-xrc42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:33.763523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.763480 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm"] Oct 30 02:35:33.764212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.764189 5998 cpu_manager.go:414] "containerMap: removing container" podUID="d1b71a2d-53c4-4e41-85ff-b82b82292598" containerName="istio-proxy" Oct 30 02:35:33.764212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.764211 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b71a2d-53c4-4e41-85ff-b82b82292598" containerName="istio-proxy" Oct 30 02:35:33.764421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.764236 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c13ee6d9-ce09-49b9-830f-5b8b25506581" containerName="istio-proxy" Oct 30 02:35:33.764421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.764243 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13ee6d9-ce09-49b9-830f-5b8b25506581" containerName="istio-proxy" Oct 30 02:35:33.764421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.764261 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a10a5bfc-b78e-4075-b194-71c14f8e0a80" containerName="revoke-keys" Oct 30 02:35:33.764421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.764268 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10a5bfc-b78e-4075-b194-71c14f8e0a80" containerName="revoke-keys" Oct 30 02:35:33.764421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.764297 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1fd3f37a-562d-4b69-b061-7c26891d0c43" containerName="revoke-keys" Oct 30 02:35:33.764421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.764304 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd3f37a-562d-4b69-b061-7c26891d0c43" containerName="revoke-keys" Oct 30 02:35:33.764421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.764320 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c3524261-ac9b-419e-82b9-c46404256f89" containerName="main" Oct 30 02:35:33.764421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.764328 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3524261-ac9b-419e-82b9-c46404256f89" containerName="main" Oct 30 02:35:33.764421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.764375 5998 cpu_manager.go:414] "containerMap: removing container" podUID="f2daa657-48ed-40a5-b5ab-3e1596a707ed" containerName="main" Oct 30 02:35:33.764421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.764383 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2daa657-48ed-40a5-b5ab-3e1596a707ed" containerName="main" Oct 30 02:35:33.764922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.764548 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd3f37a-562d-4b69-b061-7c26891d0c43" containerName="revoke-keys" Oct 30 02:35:33.764922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.764565 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3524261-ac9b-419e-82b9-c46404256f89" containerName="main" Oct 30 02:35:33.764922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.764584 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2daa657-48ed-40a5-b5ab-3e1596a707ed" containerName="main" Oct 30 02:35:33.764922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.764593 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1b71a2d-53c4-4e41-85ff-b82b82292598" containerName="istio-proxy" Oct 30 02:35:33.764922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.764605 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13ee6d9-ce09-49b9-830f-5b8b25506581" containerName="istio-proxy" Oct 30 02:35:33.764922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.764618 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10a5bfc-b78e-4075-b194-71c14f8e0a80" containerName="revoke-keys" Oct 30 02:35:33.768447 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.768423 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.780347 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.780320 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.780464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.780434 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.780464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.780457 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e4cea7-0d7d-46b1-810b-4567363be4eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.780615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.780589 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.780663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.780650 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.780742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.780724 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwd2r\" (UniqueName: \"kubernetes.io/projected/94e4cea7-0d7d-46b1-810b-4567363be4eb-kube-api-access-dwd2r\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.780914 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.780791 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.780914 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.780863 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.780914 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.780894 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.781115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.780958 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrc42\" (UniqueName: \"kubernetes.io/projected/f2daa657-48ed-40a5-b5ab-3e1596a707ed-kube-api-access-xrc42\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:33.781115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.781001 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2daa657-48ed-40a5-b5ab-3e1596a707ed-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:33.781115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.781023 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:33.781115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.781038 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:33.781115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.781052 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2daa657-48ed-40a5-b5ab-3e1596a707ed-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:33.785687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.785663 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs"] Oct 30 02:35:33.789945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.789921 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m"] Oct 30 02:35:33.790223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.790201 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.791792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.791770 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5dmpm\"" Oct 30 02:35:33.794852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.794823 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.797568 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.797546 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm"] Oct 30 02:35:33.824191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.816343 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m"] Oct 30 02:35:33.839330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.839293 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs"] Oct 30 02:35:33.843395 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.843360 5998 generic.go:356] "Generic (PLEG): container finished" podID="f2daa657-48ed-40a5-b5ab-3e1596a707ed" containerID="f1bc8eb987126f4f8c19eb637e88ccc35af9e2a29ca69c631debc7a2ae7390ac" exitCode=0 Oct 30 02:35:33.843568 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.843426 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" Oct 30 02:35:33.843568 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.843502 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" event={"ID":"f2daa657-48ed-40a5-b5ab-3e1596a707ed","Type":"ContainerDied","Data":"f1bc8eb987126f4f8c19eb637e88ccc35af9e2a29ca69c631debc7a2ae7390ac"} Oct 30 02:35:33.843568 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.843539 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t" event={"ID":"f2daa657-48ed-40a5-b5ab-3e1596a707ed","Type":"ContainerDied","Data":"3d034068ce3432d64012a13d9ae814e201fbc123cf32f5280382fe1d2b42d155"} Oct 30 02:35:33.843568 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.843563 5998 scope.go:122] "RemoveContainer" containerID="f1bc8eb987126f4f8c19eb637e88ccc35af9e2a29ca69c631debc7a2ae7390ac" Oct 30 02:35:33.866174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.866124 5998 scope.go:122] "RemoveContainer" containerID="f1bc8eb987126f4f8c19eb637e88ccc35af9e2a29ca69c631debc7a2ae7390ac" Oct 30 02:35:33.869035 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:35:33.868997 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1bc8eb987126f4f8c19eb637e88ccc35af9e2a29ca69c631debc7a2ae7390ac\": container with ID starting with f1bc8eb987126f4f8c19eb637e88ccc35af9e2a29ca69c631debc7a2ae7390ac not found: ID does not exist" containerID="f1bc8eb987126f4f8c19eb637e88ccc35af9e2a29ca69c631debc7a2ae7390ac" Oct 30 02:35:33.869209 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.869045 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1bc8eb987126f4f8c19eb637e88ccc35af9e2a29ca69c631debc7a2ae7390ac"} err="failed to get container status \"f1bc8eb987126f4f8c19eb637e88ccc35af9e2a29ca69c631debc7a2ae7390ac\": rpc error: code = NotFound desc = could not find container \"f1bc8eb987126f4f8c19eb637e88ccc35af9e2a29ca69c631debc7a2ae7390ac\": container with ID starting with f1bc8eb987126f4f8c19eb637e88ccc35af9e2a29ca69c631debc7a2ae7390ac not found: ID does not exist" Oct 30 02:35:33.881846 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.881815 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.882027 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.881876 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwd2r\" (UniqueName: \"kubernetes.io/projected/94e4cea7-0d7d-46b1-810b-4567363be4eb-kube-api-access-dwd2r\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.882027 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.881946 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.882200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.882031 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhm55\" (UniqueName: \"kubernetes.io/projected/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-kube-api-access-xhm55\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.882200 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.882075 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.882330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.882211 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.882330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.882296 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.882459 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.882421 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.882597 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.882538 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.882597 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.882557 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.882796 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.882772 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb7072a-7343-4122-9472-93c1a658cb02-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.882938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.882815 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb7072a-7343-4122-9472-93c1a658cb02-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.882938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.882842 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.882938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.882866 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.882938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.882937 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.883242 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.882958 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb7072a-7343-4122-9472-93c1a658cb02-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.883242 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.883020 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.883242 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.883042 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.883242 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.883172 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.883242 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.883199 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e4cea7-0d7d-46b1-810b-4567363be4eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.883520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.883248 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.883520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.883277 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.883520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.883309 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.883520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.883333 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.883520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.883370 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.883520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.883395 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.883520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.883456 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.883520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.883486 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.883520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.883512 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zgxj\" (UniqueName: \"kubernetes.io/projected/acb7072a-7343-4122-9472-93c1a658cb02-kube-api-access-7zgxj\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.884023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.883879 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.884023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.883935 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.884023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.883935 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e4cea7-0d7d-46b1-810b-4567363be4eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.885043 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.885020 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.886746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.886725 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.903695 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.903672 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.903797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.903778 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwd2r\" (UniqueName: \"kubernetes.io/projected/94e4cea7-0d7d-46b1-810b-4567363be4eb-kube-api-access-dwd2r\") pod \"maas-default-gateway-openshift-default-75d7c785f-bwczm\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:33.946424 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.946130 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t"] Oct 30 02:35:33.950852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.950815 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d207e8/test-model-d207e8-kserve-5dfc9fc475-nbx4t"] Oct 30 02:35:33.984829 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.984797 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.984992 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.984837 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.984992 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.984854 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zgxj\" (UniqueName: \"kubernetes.io/projected/acb7072a-7343-4122-9472-93c1a658cb02-kube-api-access-7zgxj\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.984992 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.984881 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.984992 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.984966 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhm55\" (UniqueName: \"kubernetes.io/projected/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-kube-api-access-xhm55\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.985249 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.985019 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.985249 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.985062 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb7072a-7343-4122-9472-93c1a658cb02-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.985249 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.985235 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb7072a-7343-4122-9472-93c1a658cb02-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.985416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.985277 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.985416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.985303 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.985416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.985317 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.985416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.985372 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.985416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.985408 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb7072a-7343-4122-9472-93c1a658cb02-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.985702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.985428 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.985702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.985445 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.985702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.985487 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.985702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.985601 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.985702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.985632 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.985702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.985659 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.986021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.985703 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.986021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.985876 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.986208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.986182 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.986359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.986275 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.986591 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.986567 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.986814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.986791 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.986960 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.986936 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.987024 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.986963 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb7072a-7343-4122-9472-93c1a658cb02-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.987231 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.987178 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.987417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.987397 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:33.988561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.988530 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.988661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.988585 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:33.989057 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:33.989040 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb7072a-7343-4122-9472-93c1a658cb02-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:34.006815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.006784 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zgxj\" (UniqueName: \"kubernetes.io/projected/acb7072a-7343-4122-9472-93c1a658cb02-kube-api-access-7zgxj\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:34.008273 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.008243 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhm55\" (UniqueName: \"kubernetes.io/projected/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-kube-api-access-xhm55\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:34.008371 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.008316 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb7072a-7343-4122-9472-93c1a658cb02-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-m9q2m\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:34.008489 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.008466 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-2qvfs\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:34.083571 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.083528 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:34.127965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.127918 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:34.143550 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.143265 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:34.418602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.418573 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm"] Oct 30 02:35:34.422291 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:35:34.422257 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e4cea7_0d7d_46b1_810b_4567363be4eb.slice/crio-97eb5b310e2bf74c004f9f1e134c9803303d63d5820e6bec4de14b090eea1c38 WatchSource:0}: Error finding container 97eb5b310e2bf74c004f9f1e134c9803303d63d5820e6bec4de14b090eea1c38: Status 404 returned error can't find the container with id 97eb5b310e2bf74c004f9f1e134c9803303d63d5820e6bec4de14b090eea1c38 Oct 30 02:35:34.426187 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.426150 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:35:34.426279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.426225 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:35:34.426279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.426254 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:35:34.469289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.469259 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs"] Oct 30 02:35:34.474567 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:35:34.474535 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ed4ad2_da4e_4ed2_bebe_f64b80ea093b.slice/crio-2b792f944f0b79a1b4d5c2767c1d16a858ec4dbd36b42542fea22b56ae484f41 WatchSource:0}: Error finding container 2b792f944f0b79a1b4d5c2767c1d16a858ec4dbd36b42542fea22b56ae484f41: Status 404 returned error can't find the container with id 2b792f944f0b79a1b4d5c2767c1d16a858ec4dbd36b42542fea22b56ae484f41 Oct 30 02:35:34.478690 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.478654 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:35:34.478777 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.478730 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:35:34.478777 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.478762 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:35:34.487876 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.487854 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m"] Oct 30 02:35:34.495728 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:35:34.495703 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb7072a_7343_4122_9472_93c1a658cb02.slice/crio-006656527df8a1bb3121f86fb68b46b40ed383e2e6a2c3561aa0c1c7b963d27d WatchSource:0}: Error finding container 006656527df8a1bb3121f86fb68b46b40ed383e2e6a2c3561aa0c1c7b963d27d: Status 404 returned error can't find the container with id 006656527df8a1bb3121f86fb68b46b40ed383e2e6a2c3561aa0c1c7b963d27d Oct 30 02:35:34.499199 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.499157 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:35:34.499297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.499245 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:35:34.499354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.499295 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:35:34.547792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.547760 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2daa657-48ed-40a5-b5ab-3e1596a707ed" path="/var/lib/kubelet/pods/f2daa657-48ed-40a5-b5ab-3e1596a707ed/volumes" Oct 30 02:35:34.608116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.606916 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:34.865219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.865175 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" event={"ID":"acb7072a-7343-4122-9472-93c1a658cb02","Type":"ContainerStarted","Data":"8b0c1bb4a6b32938478201745bc90f2d0ffbcfe624caf4f6a1be410f982de16b"} Oct 30 02:35:34.865517 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.865491 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" event={"ID":"acb7072a-7343-4122-9472-93c1a658cb02","Type":"ContainerStarted","Data":"006656527df8a1bb3121f86fb68b46b40ed383e2e6a2c3561aa0c1c7b963d27d"} Oct 30 02:35:34.870715 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.870522 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" event={"ID":"94e4cea7-0d7d-46b1-810b-4567363be4eb","Type":"ContainerStarted","Data":"dd9f169eaba798b00bc61f97c0fdb48011ba289ff77db7b62c63c688eb87fcc5"} Oct 30 02:35:34.871017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.870993 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" event={"ID":"94e4cea7-0d7d-46b1-810b-4567363be4eb","Type":"ContainerStarted","Data":"97eb5b310e2bf74c004f9f1e134c9803303d63d5820e6bec4de14b090eea1c38"} Oct 30 02:35:34.873254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.872570 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" event={"ID":"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b","Type":"ContainerStarted","Data":"0470cb631abc84e26e55ca4d39d66bc5d8c35568914bb2540fdb1346db1d4588"} Oct 30 02:35:34.873254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.872618 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" event={"ID":"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b","Type":"ContainerStarted","Data":"2b792f944f0b79a1b4d5c2767c1d16a858ec4dbd36b42542fea22b56ae484f41"} Oct 30 02:35:34.911496 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.911436 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" podStartSLOduration=1.911415677 podStartE2EDuration="1.911415677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:35:34.909313777 +0000 UTC m=+3964.488104870" watchObservedRunningTime="2026-10-30 02:35:34.911415677 +0000 UTC m=+3964.490206760" Oct 30 02:35:34.957034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:34.956968 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" podStartSLOduration=1.9569488160000001 podStartE2EDuration="1.956948816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:35:34.952331454 +0000 UTC m=+3964.531122536" watchObservedRunningTime="2026-10-30 02:35:34.956948816 +0000 UTC m=+3964.535739901" Oct 30 02:35:35.012109 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:35.006901 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" podStartSLOduration=2.006878 podStartE2EDuration="2.006878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:35:34.994670986 +0000 UTC m=+3964.573462072" watchObservedRunningTime="2026-10-30 02:35:35.006878 +0000 UTC m=+3964.585669081" Oct 30 02:35:35.084504 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:35.084461 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:35.085649 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:35.085615 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bwczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 02:35:35.085759 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:35.085679 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" podUID="94e4cea7-0d7d-46b1-810b-4567363be4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 02:35:35.129164 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:35.129058 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:35.130970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:35.130931 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2qvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body= Oct 30 02:35:35.131186 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:35.131011 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" podUID="c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" Oct 30 02:35:35.143702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:35.143661 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:35.145230 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:35.145202 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-m9q2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 02:35:35.145316 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:35.145261 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" podUID="acb7072a-7343-4122-9472-93c1a658cb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 02:35:35.167452 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:35.167415 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:36.084682 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:36.084645 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bwczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 02:35:36.085135 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:36.084701 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" podUID="94e4cea7-0d7d-46b1-810b-4567363be4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 02:35:36.139114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:36.134483 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2qvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body= Oct 30 02:35:36.139114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:36.134583 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" podUID="c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" Oct 30 02:35:36.148116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:36.145466 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-m9q2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 02:35:36.148116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:36.145547 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" podUID="acb7072a-7343-4122-9472-93c1a658cb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 02:35:36.893060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:36.893017 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:37.088140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:37.087531 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bwczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 02:35:37.088140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:37.087600 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" podUID="94e4cea7-0d7d-46b1-810b-4567363be4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 02:35:37.129126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:37.128570 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2qvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body= Oct 30 02:35:37.129126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:37.128632 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" podUID="c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" Oct 30 02:35:37.148171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:37.147432 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-m9q2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 02:35:37.148171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:37.147496 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" podUID="acb7072a-7343-4122-9472-93c1a658cb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 02:35:38.084544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:38.084496 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bwczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 02:35:38.084761 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:38.084560 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" podUID="94e4cea7-0d7d-46b1-810b-4567363be4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 02:35:38.134601 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:38.134560 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2qvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body= Oct 30 02:35:38.135264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:38.135222 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" podUID="c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" Oct 30 02:35:38.145568 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:38.145530 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-m9q2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 02:35:38.145827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:38.145798 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" podUID="acb7072a-7343-4122-9472-93c1a658cb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 02:35:38.573647 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:38.573609 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:39.086188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:39.086142 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bwczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 02:35:39.086393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:39.086205 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" podUID="94e4cea7-0d7d-46b1-810b-4567363be4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 02:35:39.140116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:39.131489 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2qvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body= Oct 30 02:35:39.140116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:39.131556 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" podUID="c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" Oct 30 02:35:39.163115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:39.145975 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-m9q2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 02:35:39.163115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:39.146036 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" podUID="acb7072a-7343-4122-9472-93c1a658cb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 02:35:40.084674 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:40.084634 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bwczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 02:35:40.084858 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:40.084691 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" podUID="94e4cea7-0d7d-46b1-810b-4567363be4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 02:35:40.140603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:40.140533 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2qvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body= Oct 30 02:35:40.141210 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:40.140608 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" podUID="c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" Oct 30 02:35:40.155127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:40.154500 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-m9q2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 02:35:40.155127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:40.154567 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" podUID="acb7072a-7343-4122-9472-93c1a658cb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 02:35:40.222114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:40.215027 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:40.372222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:40.359398 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:41.084446 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:41.084410 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bwczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 02:35:41.084685 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:41.084465 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" podUID="94e4cea7-0d7d-46b1-810b-4567363be4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 02:35:41.128982 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:41.128938 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2qvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body= Oct 30 02:35:41.129221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:41.129008 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" podUID="c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" Oct 30 02:35:41.144603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:41.144565 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-m9q2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 02:35:41.145270 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:41.145218 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" podUID="acb7072a-7343-4122-9472-93c1a658cb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 02:35:41.972683 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:41.972587 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:42.083984 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:42.083933 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bwczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 02:35:42.084202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:42.083999 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" podUID="94e4cea7-0d7d-46b1-810b-4567363be4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 02:35:42.128668 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:42.128626 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2qvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body= Oct 30 02:35:42.128979 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:42.128940 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" podUID="c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" Oct 30 02:35:42.144378 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:42.144343 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-m9q2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 02:35:42.144528 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:42.144422 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" podUID="acb7072a-7343-4122-9472-93c1a658cb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 02:35:43.489397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:43.489354 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:44.127310 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:44.127252 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-bwczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:35:44.127555 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:44.127337 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" podUID="94e4cea7-0d7d-46b1-810b-4567363be4eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:35:44.143256 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:44.143188 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-2qvfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:35:44.143614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:44.143534 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" podUID="c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:35:44.151274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:44.151225 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-m9q2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:35:44.151451 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:44.151289 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" podUID="acb7072a-7343-4122-9472-93c1a658cb02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:35:44.187005 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:44.183695 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:44.187005 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:44.185375 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:44.187005 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:44.185756 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:44.187005 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:44.186854 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:44.187407 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:44.187054 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:35:44.187609 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:44.187581 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:35:44.233549 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:44.233514 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:44.235336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:44.235314 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:44.236569 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:44.236552 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:35:44.870742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:44.870694 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:45.587781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:45.587746 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:47.109161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:47.109122 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:48.631069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:48.631035 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:48.854332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:48.854294 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn"] Oct 30 02:35:48.858620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:48.858591 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:48.897171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:48.897064 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn"] Oct 30 02:35:48.933705 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:48.933652 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:48.933705 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:48.933705 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:48.933995 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:48.933855 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:48.933995 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:48.933977 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:48.934142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:48.933998 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:48.934142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:48.934018 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:48.934142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:48.934041 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:48.934413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:48.934243 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l47m\" (UniqueName: \"kubernetes.io/projected/42fa7bf8-25e0-4bd8-b751-243b12d986b3-kube-api-access-5l47m\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:48.934413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:48.934287 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.035747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.035698 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.035941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.035765 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.035941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.035865 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.036067 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.036006 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.036067 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.036035 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.036218 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.036068 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.036218 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.036118 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.036218 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.036153 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.036218 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.036181 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l47m\" (UniqueName: \"kubernetes.io/projected/42fa7bf8-25e0-4bd8-b751-243b12d986b3-kube-api-access-5l47m\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.036218 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.036210 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.036505 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.036182 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.036505 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.036350 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.036633 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.036518 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.036750 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.036718 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.038560 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.038536 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.038991 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.038969 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.059290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.059258 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.060513 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.060484 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l47m\" (UniqueName: \"kubernetes.io/projected/42fa7bf8-25e0-4bd8-b751-243b12d986b3-kube-api-access-5l47m\") pod \"maas-default-gateway-openshift-default-75d7c785f-mzxsn\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.153753 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.153658 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:49.179127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.179079 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:49.392201 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.392166 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:49.546409 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.546378 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn"] Oct 30 02:35:49.548768 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:35:49.548737 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42fa7bf8_25e0_4bd8_b751_243b12d986b3.slice/crio-4489792261eb1a3e27a8ca3c5e5fbccdf7c9e375ad9ace63c3243fbc66a342ae WatchSource:0}: Error finding container 4489792261eb1a3e27a8ca3c5e5fbccdf7c9e375ad9ace63c3243fbc66a342ae: Status 404 returned error can't find the container with id 4489792261eb1a3e27a8ca3c5e5fbccdf7c9e375ad9ace63c3243fbc66a342ae Oct 30 02:35:49.552313 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.552278 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:35:49.552404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.552350 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:35:49.552404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.552378 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:35:49.868998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:49.868905 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:50.135477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:50.135363 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" event={"ID":"42fa7bf8-25e0-4bd8-b751-243b12d986b3","Type":"ContainerStarted","Data":"4b7c66f045a1d2292dd404ae604ee47c603d9e0acd3a591c7aa5f3243f2a2dd9"} Oct 30 02:35:50.135477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:50.135440 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" event={"ID":"42fa7bf8-25e0-4bd8-b751-243b12d986b3","Type":"ContainerStarted","Data":"4489792261eb1a3e27a8ca3c5e5fbccdf7c9e375ad9ace63c3243fbc66a342ae"} Oct 30 02:35:50.179744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:50.179705 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:50.181105 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:50.181059 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mzxsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Oct 30 02:35:50.181258 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:50.181164 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" podUID="42fa7bf8-25e0-4bd8-b751-243b12d986b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Oct 30 02:35:50.217110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:50.212019 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" podStartSLOduration=2.212000363 podStartE2EDuration="2.212000363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:35:50.201291123 +0000 UTC m=+3979.780082205" watchObservedRunningTime="2026-10-30 02:35:50.212000363 +0000 UTC m=+3979.790791445" Oct 30 02:35:50.242112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:50.236463 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m"] Oct 30 02:35:50.242112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:50.236709 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" podUID="025cc8ef-572d-4f6a-851c-2e59fb797189" containerName="istio-proxy" containerID="cri-o://2e037cefb4108dd31ce72b75e2897b437490810ae7f5efb2c17123773e729e18" gracePeriod=30 Oct 30 02:35:50.273799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:50.273760 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:50.467112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:50.456388 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:51.180242 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:51.180208 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mzxsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Oct 30 02:35:51.180662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:51.180259 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" podUID="42fa7bf8-25e0-4bd8-b751-243b12d986b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Oct 30 02:35:51.195193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:51.195164 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:51.327166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:51.327049 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:51.448669 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:51.448580 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:51.962629 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:51.962583 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:52.085172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:52.085128 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:52.197297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:52.197243 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mzxsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Oct 30 02:35:52.197919 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:52.197864 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" podUID="42fa7bf8-25e0-4bd8-b751-243b12d986b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Oct 30 02:35:52.726243 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:52.709620 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:53.179926 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:53.179894 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mzxsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Oct 30 02:35:53.180139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:53.179964 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" podUID="42fa7bf8-25e0-4bd8-b751-243b12d986b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Oct 30 02:35:54.180230 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:54.180194 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mzxsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Oct 30 02:35:54.180655 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:54.180257 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" podUID="42fa7bf8-25e0-4bd8-b751-243b12d986b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Oct 30 02:35:54.298740 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:54.298705 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:55.179439 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.179399 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mzxsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Oct 30 02:35:55.179660 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.179487 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" podUID="42fa7bf8-25e0-4bd8-b751-243b12d986b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Oct 30 02:35:55.249130 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:35:55.249069 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:35:55.590184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.590146 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:35:55.675032 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.674993 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-credential-socket\") pod \"025cc8ef-572d-4f6a-851c-2e59fb797189\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " Oct 30 02:35:55.675215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.675073 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-envoy\") pod \"025cc8ef-572d-4f6a-851c-2e59fb797189\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " Oct 30 02:35:55.675215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.675118 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-workload-socket\") pod \"025cc8ef-572d-4f6a-851c-2e59fb797189\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " Oct 30 02:35:55.675215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.675138 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/025cc8ef-572d-4f6a-851c-2e59fb797189-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025cc8ef-572d-4f6a-851c-2e59fb797189-istiod-ca-cert\") pod \"025cc8ef-572d-4f6a-851c-2e59fb797189\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " Oct 30 02:35:55.675215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.675192 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-podinfo\") pod \"025cc8ef-572d-4f6a-851c-2e59fb797189\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " Oct 30 02:35:55.675392 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.675239 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025cc8ef-572d-4f6a-851c-2e59fb797189-kube-api-access-2htds\" (UniqueName: \"kubernetes.io/projected/025cc8ef-572d-4f6a-851c-2e59fb797189-kube-api-access-2htds\") pod \"025cc8ef-572d-4f6a-851c-2e59fb797189\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " Oct 30 02:35:55.675392 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.675279 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-data\") pod \"025cc8ef-572d-4f6a-851c-2e59fb797189\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " Oct 30 02:35:55.675392 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.675304 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-workload-certs\") pod \"025cc8ef-572d-4f6a-851c-2e59fb797189\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " Oct 30 02:35:55.675741 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.675708 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-workload-socket" pod "025cc8ef-572d-4f6a-851c-2e59fb797189" (UID: "025cc8ef-572d-4f6a-851c-2e59fb797189"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:55.676174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.676147 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-data" pod "025cc8ef-572d-4f6a-851c-2e59fb797189" (UID: "025cc8ef-572d-4f6a-851c-2e59fb797189"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:55.676331 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.676300 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-credential-socket" pod "025cc8ef-572d-4f6a-851c-2e59fb797189" (UID: "025cc8ef-572d-4f6a-851c-2e59fb797189"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:55.676419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.676315 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025cc8ef-572d-4f6a-851c-2e59fb797189-istiod-ca-cert" pod "025cc8ef-572d-4f6a-851c-2e59fb797189" (UID: "025cc8ef-572d-4f6a-851c-2e59fb797189"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:35:55.676523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.676504 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-token\" (UniqueName: \"kubernetes.io/projected/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-token\") pod \"025cc8ef-572d-4f6a-851c-2e59fb797189\" (UID: \"025cc8ef-572d-4f6a-851c-2e59fb797189\") " Oct 30 02:35:55.676647 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.676610 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-workload-certs" pod "025cc8ef-572d-4f6a-851c-2e59fb797189" (UID: "025cc8ef-572d-4f6a-851c-2e59fb797189"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:55.677193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.677170 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:55.677291 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.677195 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/025cc8ef-572d-4f6a-851c-2e59fb797189-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:55.677291 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.677208 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:55.677291 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.677219 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:55.677291 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.677231 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:55.678437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.678415 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025cc8ef-572d-4f6a-851c-2e59fb797189-kube-api-access-2htds" pod "025cc8ef-572d-4f6a-851c-2e59fb797189" (UID: "025cc8ef-572d-4f6a-851c-2e59fb797189"). InnerVolumeSpecName "kube-api-access-2htds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:55.678560 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.678542 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-envoy" pod "025cc8ef-572d-4f6a-851c-2e59fb797189" (UID: "025cc8ef-572d-4f6a-851c-2e59fb797189"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:35:55.678868 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.678842 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-podinfo" pod "025cc8ef-572d-4f6a-851c-2e59fb797189" (UID: "025cc8ef-572d-4f6a-851c-2e59fb797189"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:35:55.679834 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.679811 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-token" pod "025cc8ef-572d-4f6a-851c-2e59fb797189" (UID: "025cc8ef-572d-4f6a-851c-2e59fb797189"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:35:55.778233 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.778146 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:55.778233 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.778180 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:55.778233 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.778190 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2htds\" (UniqueName: \"kubernetes.io/projected/025cc8ef-572d-4f6a-851c-2e59fb797189-kube-api-access-2htds\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:55.778233 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.778199 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/025cc8ef-572d-4f6a-851c-2e59fb797189-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:35:55.881302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:55.881264 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:56.180158 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:56.180126 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mzxsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Oct 30 02:35:56.180381 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:56.180356 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" podUID="42fa7bf8-25e0-4bd8-b751-243b12d986b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Oct 30 02:35:56.251771 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:56.251726 5998 generic.go:356] "Generic (PLEG): container finished" podID="025cc8ef-572d-4f6a-851c-2e59fb797189" containerID="2e037cefb4108dd31ce72b75e2897b437490810ae7f5efb2c17123773e729e18" exitCode=0 Oct 30 02:35:56.252255 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:56.252034 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" event={"ID":"025cc8ef-572d-4f6a-851c-2e59fb797189","Type":"ContainerDied","Data":"2e037cefb4108dd31ce72b75e2897b437490810ae7f5efb2c17123773e729e18"} Oct 30 02:35:56.252255 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:56.252113 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" event={"ID":"025cc8ef-572d-4f6a-851c-2e59fb797189","Type":"ContainerDied","Data":"adfe47df0e8760111d80357094b6478f280f4d8c080c82e4a226c00abe90c35d"} Oct 30 02:35:56.252255 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:56.252148 5998 scope.go:122] "RemoveContainer" containerID="2e037cefb4108dd31ce72b75e2897b437490810ae7f5efb2c17123773e729e18" Oct 30 02:35:56.252389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:56.252331 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m" Oct 30 02:35:56.270865 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:56.270838 5998 scope.go:122] "RemoveContainer" containerID="1cc92474d04402c331a74bccbbd597eefb521f2411ec393cecf2a3348146d7d5" Oct 30 02:35:56.282035 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:56.281948 5998 scope.go:122] "RemoveContainer" containerID="2e037cefb4108dd31ce72b75e2897b437490810ae7f5efb2c17123773e729e18" Oct 30 02:35:56.282687 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:35:56.282659 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e037cefb4108dd31ce72b75e2897b437490810ae7f5efb2c17123773e729e18\": container with ID starting with 2e037cefb4108dd31ce72b75e2897b437490810ae7f5efb2c17123773e729e18 not found: ID does not exist" containerID="2e037cefb4108dd31ce72b75e2897b437490810ae7f5efb2c17123773e729e18" Oct 30 02:35:56.282811 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:56.282695 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e037cefb4108dd31ce72b75e2897b437490810ae7f5efb2c17123773e729e18"} err="failed to get container status \"2e037cefb4108dd31ce72b75e2897b437490810ae7f5efb2c17123773e729e18\": rpc error: code = NotFound desc = could not find container \"2e037cefb4108dd31ce72b75e2897b437490810ae7f5efb2c17123773e729e18\": container with ID starting with 2e037cefb4108dd31ce72b75e2897b437490810ae7f5efb2c17123773e729e18 not found: ID does not exist" Oct 30 02:35:56.282811 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:56.282718 5998 scope.go:122] "RemoveContainer" containerID="1cc92474d04402c331a74bccbbd597eefb521f2411ec393cecf2a3348146d7d5" Oct 30 02:35:56.283042 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:35:56.283012 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cc92474d04402c331a74bccbbd597eefb521f2411ec393cecf2a3348146d7d5\": container with ID starting with 1cc92474d04402c331a74bccbbd597eefb521f2411ec393cecf2a3348146d7d5 not found: ID does not exist" containerID="1cc92474d04402c331a74bccbbd597eefb521f2411ec393cecf2a3348146d7d5" Oct 30 02:35:56.283125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:56.283053 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cc92474d04402c331a74bccbbd597eefb521f2411ec393cecf2a3348146d7d5"} err="failed to get container status \"1cc92474d04402c331a74bccbbd597eefb521f2411ec393cecf2a3348146d7d5\": rpc error: code = NotFound desc = could not find container \"1cc92474d04402c331a74bccbbd597eefb521f2411ec393cecf2a3348146d7d5\": container with ID starting with 1cc92474d04402c331a74bccbbd597eefb521f2411ec393cecf2a3348146d7d5 not found: ID does not exist" Oct 30 02:35:56.330177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:56.330136 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m"] Oct 30 02:35:56.335114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:56.333542 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4593af-openshift-default-7f586bb5b5-4jw7m"] Oct 30 02:35:56.542006 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:56.541910 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="025cc8ef-572d-4f6a-851c-2e59fb797189" path="/var/lib/kubelet/pods/025cc8ef-572d-4f6a-851c-2e59fb797189/volumes" Oct 30 02:35:57.179869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:57.179832 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mzxsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Oct 30 02:35:57.180084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:57.179894 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" podUID="42fa7bf8-25e0-4bd8-b751-243b12d986b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Oct 30 02:35:57.402883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:57.402842 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:58.180050 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:58.180006 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-mzxsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Oct 30 02:35:58.180318 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:58.180074 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" podUID="42fa7bf8-25e0-4bd8-b751-243b12d986b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Oct 30 02:35:59.063257 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:59.063222 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:35:59.186444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:59.186408 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:59.186898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:59.186871 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:35:59.187726 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:35:59.187701 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:36:00.617310 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:00.617258 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:01.105642 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:01.105603 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:01.212821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:01.212788 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:01.743759 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:01.743717 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:02.108458 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:02.108356 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:02.271996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:02.271958 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:02.881534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:02.881501 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:03.437958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:03.437919 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:36:03.927585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:03.927553 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:05.729419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:05.729374 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:07.379029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:07.378965 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:08.883591 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:08.883550 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:10.045728 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:10.045679 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:10.327206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:10.327117 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:10.455113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:10.455038 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:10.603743 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:10.603644 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:11.459528 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.459488 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t"] Oct 30 02:36:11.460377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.460351 5998 cpu_manager.go:414] "containerMap: removing container" podUID="025cc8ef-572d-4f6a-851c-2e59fb797189" containerName="istio-proxy" Oct 30 02:36:11.460377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.460378 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025cc8ef-572d-4f6a-851c-2e59fb797189" containerName="istio-proxy" Oct 30 02:36:11.460595 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.460578 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="025cc8ef-572d-4f6a-851c-2e59fb797189" containerName="istio-proxy" Oct 30 02:36:11.464080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.464057 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t" Oct 30 02:36:11.478830 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.478810 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:36:11.483692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.483669 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t"] Oct 30 02:36:11.489702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.489677 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:36:11.489804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.489738 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:36:11.489860 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.489848 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:36:11.603610 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.603566 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e352468-3606-467c-abd3-4654142859b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfcc483a-6ss2t\" (UID: \"7e352468-3606-467c-abd3-4654142859b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t" Oct 30 02:36:11.603893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.603866 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e352468-3606-467c-abd3-4654142859b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfcc483a-6ss2t\" (UID: \"7e352468-3606-467c-abd3-4654142859b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t" Oct 30 02:36:11.604216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.604185 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dst9r\" (UniqueName: \"kubernetes.io/projected/7e352468-3606-467c-abd3-4654142859b0-kube-api-access-dst9r\") pod \"maas-api-revoke-sub-dfcc483a-6ss2t\" (UID: \"7e352468-3606-467c-abd3-4654142859b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t" Oct 30 02:36:11.705909 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.705863 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e352468-3606-467c-abd3-4654142859b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfcc483a-6ss2t\" (UID: \"7e352468-3606-467c-abd3-4654142859b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t" Oct 30 02:36:11.706107 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.706077 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dst9r\" (UniqueName: \"kubernetes.io/projected/7e352468-3606-467c-abd3-4654142859b0-kube-api-access-dst9r\") pod \"maas-api-revoke-sub-dfcc483a-6ss2t\" (UID: \"7e352468-3606-467c-abd3-4654142859b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t" Oct 30 02:36:11.706194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.706154 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e352468-3606-467c-abd3-4654142859b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfcc483a-6ss2t\" (UID: \"7e352468-3606-467c-abd3-4654142859b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t" Oct 30 02:36:11.706762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.706733 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e352468-3606-467c-abd3-4654142859b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfcc483a-6ss2t\" (UID: \"7e352468-3606-467c-abd3-4654142859b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t" Oct 30 02:36:11.708753 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.708728 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e352468-3606-467c-abd3-4654142859b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfcc483a-6ss2t\" (UID: \"7e352468-3606-467c-abd3-4654142859b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t" Oct 30 02:36:11.728515 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.728444 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dst9r\" (UniqueName: \"kubernetes.io/projected/7e352468-3606-467c-abd3-4654142859b0-kube-api-access-dst9r\") pod \"maas-api-revoke-sub-dfcc483a-6ss2t\" (UID: \"7e352468-3606-467c-abd3-4654142859b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t" Oct 30 02:36:11.778283 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:11.778242 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t" Oct 30 02:36:12.089425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:12.089394 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t"] Oct 30 02:36:12.092029 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:36:12.091998 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e352468_3606_467c_abd3_4654142859b0.slice/crio-361ceb60f7a14c02c85e3bba5194e9efeb684f5ca31ff8cd9104be6e66bf42a5 WatchSource:0}: Error finding container 361ceb60f7a14c02c85e3bba5194e9efeb684f5ca31ff8cd9104be6e66bf42a5: Status 404 returned error can't find the container with id 361ceb60f7a14c02c85e3bba5194e9efeb684f5ca31ff8cd9104be6e66bf42a5 Oct 30 02:36:12.331853 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:12.331819 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:12.426618 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:12.426576 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t" event={"ID":"7e352468-3606-467c-abd3-4654142859b0","Type":"ContainerStarted","Data":"361ceb60f7a14c02c85e3bba5194e9efeb684f5ca31ff8cd9104be6e66bf42a5"} Oct 30 02:36:13.438223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:13.438182 5998 generic.go:356] "Generic (PLEG): container finished" podID="7e352468-3606-467c-abd3-4654142859b0" containerID="83a8694141a8d221fd73bbedce7176445c1fa62e41e25313d8b6e683d4d3b677" exitCode=0 Oct 30 02:36:13.438742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:13.438264 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t" event={"ID":"7e352468-3606-467c-abd3-4654142859b0","Type":"ContainerDied","Data":"83a8694141a8d221fd73bbedce7176445c1fa62e41e25313d8b6e683d4d3b677"} Oct 30 02:36:13.719867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:13.719779 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:14.458506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:14.458476 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:14.671919 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:14.671891 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t" Oct 30 02:36:14.781582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:14.781486 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e352468-3606-467c-abd3-4654142859b0-kube-api-access-dst9r\" (UniqueName: \"kubernetes.io/projected/7e352468-3606-467c-abd3-4654142859b0-kube-api-access-dst9r\") pod \"7e352468-3606-467c-abd3-4654142859b0\" (UID: \"7e352468-3606-467c-abd3-4654142859b0\") " Oct 30 02:36:14.781832 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:14.781810 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e352468-3606-467c-abd3-4654142859b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e352468-3606-467c-abd3-4654142859b0-maas-api-openshift-ca\") pod \"7e352468-3606-467c-abd3-4654142859b0\" (UID: \"7e352468-3606-467c-abd3-4654142859b0\") " Oct 30 02:36:14.781900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:14.781854 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e352468-3606-467c-abd3-4654142859b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e352468-3606-467c-abd3-4654142859b0-maas-api-xks-ca\") pod \"7e352468-3606-467c-abd3-4654142859b0\" (UID: \"7e352468-3606-467c-abd3-4654142859b0\") " Oct 30 02:36:14.782276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:14.782244 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e352468-3606-467c-abd3-4654142859b0-maas-api-openshift-ca" pod "7e352468-3606-467c-abd3-4654142859b0" (UID: "7e352468-3606-467c-abd3-4654142859b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:36:14.782588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:14.782567 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e352468-3606-467c-abd3-4654142859b0-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:36:14.783800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:14.783776 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e352468-3606-467c-abd3-4654142859b0-kube-api-access-dst9r" pod "7e352468-3606-467c-abd3-4654142859b0" (UID: "7e352468-3606-467c-abd3-4654142859b0"). InnerVolumeSpecName "kube-api-access-dst9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:36:14.784038 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:14.784018 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e352468-3606-467c-abd3-4654142859b0-maas-api-xks-ca" pod "7e352468-3606-467c-abd3-4654142859b0" (UID: "7e352468-3606-467c-abd3-4654142859b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:36:14.884457 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:14.884423 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e352468-3606-467c-abd3-4654142859b0-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:36:14.884457 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:14.884453 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dst9r\" (UniqueName: \"kubernetes.io/projected/7e352468-3606-467c-abd3-4654142859b0-kube-api-access-dst9r\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:36:15.456370 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:15.456341 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t" Oct 30 02:36:15.456572 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:15.456387 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t" event={"ID":"7e352468-3606-467c-abd3-4654142859b0","Type":"ContainerDied","Data":"361ceb60f7a14c02c85e3bba5194e9efeb684f5ca31ff8cd9104be6e66bf42a5"} Oct 30 02:36:15.456572 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:15.456440 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="361ceb60f7a14c02c85e3bba5194e9efeb684f5ca31ff8cd9104be6e66bf42a5" Oct 30 02:36:15.948072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:15.948036 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:16.528644 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:16.528605 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:17.509694 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:17.509646 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:18.114487 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:18.114433 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:19.124062 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:19.124026 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:20.823492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:20.823432 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:20.984764 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:20.984706 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:21.615511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:21.615417 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:21.837482 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:21.837441 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:22.007149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:22.007110 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq"] Oct 30 02:36:22.007685 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:22.007652 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" podUID="2fd2791e-dcdd-40a2-be03-95c5c266d99c" containerName="istio-proxy" containerID="cri-o://31427e52ef612d3ab97df4e30432e98ecf649b426c1a43e9a6da808ae6a0a480" gracePeriod=30 Oct 30 02:36:22.102304 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:22.102268 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:22.800390 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:22.800353 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:22.945856 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:22.945814 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:23.153005 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:23.152876 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:24.592986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:24.592951 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:25.219191 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:36:25.219152 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:36:25.684589 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:25.684545 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:26.180023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:26.179989 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:27.293968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.293938 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:36:27.400884 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.400847 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd2791e-dcdd-40a2-be03-95c5c266d99c-kube-api-access-r5kqm\" (UniqueName: \"kubernetes.io/projected/2fd2791e-dcdd-40a2-be03-95c5c266d99c-kube-api-access-r5kqm\") pod \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " Oct 30 02:36:27.400884 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.400892 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-workload-socket\") pod \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " Oct 30 02:36:27.401183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.401036 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-workload-certs\") pod \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " Oct 30 02:36:27.401183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.401078 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-podinfo\") pod \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " Oct 30 02:36:27.401183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.401142 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-envoy\") pod \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " Oct 30 02:36:27.401183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.401180 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-credential-socket\") pod \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " Oct 30 02:36:27.401404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.401220 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istiod-ca-cert\") pod \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " Oct 30 02:36:27.401404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.401291 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-token\" (UniqueName: \"kubernetes.io/projected/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-token\") pod \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " Oct 30 02:36:27.401404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.401322 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-data\") pod \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\" (UID: \"2fd2791e-dcdd-40a2-be03-95c5c266d99c\") " Oct 30 02:36:27.401404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.401318 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-workload-socket" pod "2fd2791e-dcdd-40a2-be03-95c5c266d99c" (UID: "2fd2791e-dcdd-40a2-be03-95c5c266d99c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:36:27.401404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.401370 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-workload-certs" pod "2fd2791e-dcdd-40a2-be03-95c5c266d99c" (UID: "2fd2791e-dcdd-40a2-be03-95c5c266d99c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:36:27.401678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.401556 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-credential-socket" pod "2fd2791e-dcdd-40a2-be03-95c5c266d99c" (UID: "2fd2791e-dcdd-40a2-be03-95c5c266d99c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:36:27.402130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.402068 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:36:27.402643 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.402623 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:36:27.402807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.402792 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:36:27.402919 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.402555 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-data" pod "2fd2791e-dcdd-40a2-be03-95c5c266d99c" (UID: "2fd2791e-dcdd-40a2-be03-95c5c266d99c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:36:27.403028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.402713 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istiod-ca-cert" pod "2fd2791e-dcdd-40a2-be03-95c5c266d99c" (UID: "2fd2791e-dcdd-40a2-be03-95c5c266d99c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:36:27.403907 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.403882 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd2791e-dcdd-40a2-be03-95c5c266d99c-kube-api-access-r5kqm" pod "2fd2791e-dcdd-40a2-be03-95c5c266d99c" (UID: "2fd2791e-dcdd-40a2-be03-95c5c266d99c"). InnerVolumeSpecName "kube-api-access-r5kqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:36:27.403990 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.403932 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-token" pod "2fd2791e-dcdd-40a2-be03-95c5c266d99c" (UID: "2fd2791e-dcdd-40a2-be03-95c5c266d99c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:36:27.404040 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.403987 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-podinfo" pod "2fd2791e-dcdd-40a2-be03-95c5c266d99c" (UID: "2fd2791e-dcdd-40a2-be03-95c5c266d99c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:36:27.404077 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.404051 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-envoy" pod "2fd2791e-dcdd-40a2-be03-95c5c266d99c" (UID: "2fd2791e-dcdd-40a2-be03-95c5c266d99c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:36:27.503940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.503892 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5kqm\" (UniqueName: \"kubernetes.io/projected/2fd2791e-dcdd-40a2-be03-95c5c266d99c-kube-api-access-r5kqm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:36:27.503940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.503932 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:36:27.503940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.503945 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:36:27.504276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.503957 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:36:27.504276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.503968 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:36:27.504276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.503979 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fd2791e-dcdd-40a2-be03-95c5c266d99c-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:36:27.557157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.557057 5998 generic.go:356] "Generic (PLEG): container finished" podID="2fd2791e-dcdd-40a2-be03-95c5c266d99c" containerID="31427e52ef612d3ab97df4e30432e98ecf649b426c1a43e9a6da808ae6a0a480" exitCode=0 Oct 30 02:36:27.557157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.557127 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" event={"ID":"2fd2791e-dcdd-40a2-be03-95c5c266d99c","Type":"ContainerDied","Data":"31427e52ef612d3ab97df4e30432e98ecf649b426c1a43e9a6da808ae6a0a480"} Oct 30 02:36:27.557367 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.557184 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" event={"ID":"2fd2791e-dcdd-40a2-be03-95c5c266d99c","Type":"ContainerDied","Data":"1c9f2909b541414bd7a89fcc09ba180d068a9868cec524ea4b972cc7dea468d5"} Oct 30 02:36:27.557367 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.557207 5998 scope.go:122] "RemoveContainer" containerID="31427e52ef612d3ab97df4e30432e98ecf649b426c1a43e9a6da808ae6a0a480" Oct 30 02:36:27.557367 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.557134 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq" Oct 30 02:36:27.569744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.569718 5998 scope.go:122] "RemoveContainer" containerID="deaf88684552f21e3be7d70c5b86b55003114083a61b7d565382df82428faad2" Oct 30 02:36:27.580935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.580908 5998 scope.go:122] "RemoveContainer" containerID="31427e52ef612d3ab97df4e30432e98ecf649b426c1a43e9a6da808ae6a0a480" Oct 30 02:36:27.581337 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:36:27.581307 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31427e52ef612d3ab97df4e30432e98ecf649b426c1a43e9a6da808ae6a0a480\": container with ID starting with 31427e52ef612d3ab97df4e30432e98ecf649b426c1a43e9a6da808ae6a0a480 not found: ID does not exist" containerID="31427e52ef612d3ab97df4e30432e98ecf649b426c1a43e9a6da808ae6a0a480" Oct 30 02:36:27.581453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.581353 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31427e52ef612d3ab97df4e30432e98ecf649b426c1a43e9a6da808ae6a0a480"} err="failed to get container status \"31427e52ef612d3ab97df4e30432e98ecf649b426c1a43e9a6da808ae6a0a480\": rpc error: code = NotFound desc = could not find container \"31427e52ef612d3ab97df4e30432e98ecf649b426c1a43e9a6da808ae6a0a480\": container with ID starting with 31427e52ef612d3ab97df4e30432e98ecf649b426c1a43e9a6da808ae6a0a480 not found: ID does not exist" Oct 30 02:36:27.581453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.581380 5998 scope.go:122] "RemoveContainer" containerID="deaf88684552f21e3be7d70c5b86b55003114083a61b7d565382df82428faad2" Oct 30 02:36:27.581714 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:36:27.581667 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deaf88684552f21e3be7d70c5b86b55003114083a61b7d565382df82428faad2\": container with ID starting with deaf88684552f21e3be7d70c5b86b55003114083a61b7d565382df82428faad2 not found: ID does not exist" containerID="deaf88684552f21e3be7d70c5b86b55003114083a61b7d565382df82428faad2" Oct 30 02:36:27.581805 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.581724 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deaf88684552f21e3be7d70c5b86b55003114083a61b7d565382df82428faad2"} err="failed to get container status \"deaf88684552f21e3be7d70c5b86b55003114083a61b7d565382df82428faad2\": rpc error: code = NotFound desc = could not find container \"deaf88684552f21e3be7d70c5b86b55003114083a61b7d565382df82428faad2\": container with ID starting with deaf88684552f21e3be7d70c5b86b55003114083a61b7d565382df82428faad2 not found: ID does not exist" Oct 30 02:36:27.607653 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.607619 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq"] Oct 30 02:36:27.616640 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.616610 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-db01c4-openshift-default-779c66d87-jrxqq"] Oct 30 02:36:27.687501 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:27.687465 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:28.485309 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:28.480250 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:28.541679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:28.541631 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fd2791e-dcdd-40a2-be03-95c5c266d99c" path="/var/lib/kubelet/pods/2fd2791e-dcdd-40a2-be03-95c5c266d99c/volumes" Oct 30 02:36:28.612365 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:28.612303 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:28.891374 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:28.891275 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:29.586371 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:29.586334 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:30.288707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:30.288648 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:31.164352 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:31.164320 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:32.862400 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:32.862314 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:34.413336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:34.413296 5998 scope.go:122] "RemoveContainer" containerID="f3103f1f591bf929c011d8663f6c90fb37f46176b7d3b7bbfb8f9ba956a99290" Oct 30 02:36:34.488382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:34.488329 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:35.823955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:35.823917 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:36.006425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:36.006381 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:36.911768 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:36.911726 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:37.669298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:37.669256 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:39.285542 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:39.285499 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:40.360950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:40.360913 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:41.010651 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:41.010611 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:42.699544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:42.699504 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:44.382406 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:44.382373 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:45.867171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:45.867125 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:46.234594 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:46.234558 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:36:47.507518 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:47.507476 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:48.935114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:48.932775 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:49.864239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:49.863174 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" podUID="9ced29b5-0313-45a9-8a67-54e28ebc4bf2" containerName="main" probeResult="failure" output="Get \"https://10.133.1.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:36:50.236229 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:50.236169 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 02:36:50.280662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:50.280620 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:50.390916 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:50.388220 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 02:36:50.416231 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:50.416181 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 02:36:50.441058 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:50.440212 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 02:36:50.595172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:50.594276 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" podUID="76604767-dec2-42cd-b953-65221160999f" containerName="main" probeResult="failure" output="Get \"https://10.133.1.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:36:50.890175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:50.889213 5998 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded" Oct 30 02:36:50.890175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:50.889345 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" podUID="9ced29b5-0313-45a9-8a67-54e28ebc4bf2" containerName="main" probeResult="failure" output="Get \"https://10.133.1.30:8000/ready\": context deadline exceeded" Oct 30 02:36:50.890456 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:50.890336 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:36:51.282113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:51.281858 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:36:51.334115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:51.282662 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:36:51.334115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:51.283117 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 02:36:51.334115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:51.294442 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 02:36:51.405125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:51.404507 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:36:51.502115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:51.489358 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Oct 30 02:36:51.502115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:51.490907 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 02:36:51.663219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:51.637359 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:36:51.709920 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:51.709861 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" podUID="76604767-dec2-42cd-b953-65221160999f" containerName="main" probeResult="failure" output="Get \"https://10.133.1.28:8000/ready\": context deadline exceeded" Oct 30 02:36:51.710621 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:51.710598 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:36:51.748681 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:51.748621 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:51.889625 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:51.889580 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" podUID="9ced29b5-0313-45a9-8a67-54e28ebc4bf2" containerName="main" probeResult="failure" output="Get \"https://10.133.1.30:8000/ready\": context deadline exceeded" Oct 30 02:36:51.913873 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:51.913782 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 02:36:51.940399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:51.940361 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:36:51.975859 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:51.975826 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 02:36:51.989115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:51.989044 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:36:53.332959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:53.332919 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:54.458581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:54.458539 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:54.566800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:54.566760 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:55.059672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:55.059639 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:55.223316 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:36:55.223264 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:36:55.236469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:55.236435 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:55.645302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:55.645265 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:56.577255 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:56.577215 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:56.707038 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:56.707001 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:57.046796 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:57.046759 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:36:58.667821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:36:58.667782 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:00.362866 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:00.362784 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:01.968866 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:01.968827 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:03.674814 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:03.674780 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:05.282559 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:05.282517 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:06.855677 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:06.855641 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:07.452823 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:07.452785 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:08.308015 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:08.307970 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:08.494788 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:08.494735 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:09.906995 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:09.906953 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:10.683459 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:10.683416 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:12.272145 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:12.272048 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:12.788314 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:12.788277 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:12.907862 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:12.907827 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:13.458926 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:13.458891 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:13.821894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:13.821807 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb"] Oct 30 02:37:13.822402 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:13.822359 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" podUID="49169ae5-7185-44fc-9c12-de093c4a1510" containerName="istio-proxy" containerID="cri-o://3f194d72314a0a862ae405d5bc92780190597d7b448d03584e88874a95538972" gracePeriod=30 Oct 30 02:37:13.859341 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:13.859300 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:14.021147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:14.018625 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:14.769304 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:14.769262 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:14.901853 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:14.901815 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:15.735897 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:15.735858 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:15.918072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:15.918037 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:17.566276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:17.566238 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:19.100767 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.100653 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:19.110081 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.110056 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:37:19.259667 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.259629 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49169ae5-7185-44fc-9c12-de093c4a1510-kube-api-access-thbm4\" (UniqueName: \"kubernetes.io/projected/49169ae5-7185-44fc-9c12-de093c4a1510-kube-api-access-thbm4\") pod \"49169ae5-7185-44fc-9c12-de093c4a1510\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " Oct 30 02:37:19.259855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.259712 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49169ae5-7185-44fc-9c12-de093c4a1510-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49169ae5-7185-44fc-9c12-de093c4a1510-istio-podinfo\") pod \"49169ae5-7185-44fc-9c12-de093c4a1510\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " Oct 30 02:37:19.259855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.259795 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-workload-socket\") pod \"49169ae5-7185-44fc-9c12-de093c4a1510\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " Oct 30 02:37:19.259855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.259829 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-workload-certs\") pod \"49169ae5-7185-44fc-9c12-de093c4a1510\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " Oct 30 02:37:19.260041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.259878 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-credential-socket\") pod \"49169ae5-7185-44fc-9c12-de093c4a1510\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " Oct 30 02:37:19.260041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.260010 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49169ae5-7185-44fc-9c12-de093c4a1510-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49169ae5-7185-44fc-9c12-de093c4a1510-istiod-ca-cert\") pod \"49169ae5-7185-44fc-9c12-de093c4a1510\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " Oct 30 02:37:19.260217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.260045 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49169ae5-7185-44fc-9c12-de093c4a1510-istio-token\" (UniqueName: \"kubernetes.io/projected/49169ae5-7185-44fc-9c12-de093c4a1510-istio-token\") pod \"49169ae5-7185-44fc-9c12-de093c4a1510\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " Oct 30 02:37:19.260217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.260080 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-istio-data\") pod \"49169ae5-7185-44fc-9c12-de093c4a1510\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " Oct 30 02:37:19.260217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.260190 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-istio-envoy\") pod \"49169ae5-7185-44fc-9c12-de093c4a1510\" (UID: \"49169ae5-7185-44fc-9c12-de093c4a1510\") " Oct 30 02:37:19.260217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.260191 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-workload-socket" pod "49169ae5-7185-44fc-9c12-de093c4a1510" (UID: "49169ae5-7185-44fc-9c12-de093c4a1510"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:37:19.260450 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.260225 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-workload-certs" pod "49169ae5-7185-44fc-9c12-de093c4a1510" (UID: "49169ae5-7185-44fc-9c12-de093c4a1510"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:37:19.260450 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.260385 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-credential-socket" pod "49169ae5-7185-44fc-9c12-de093c4a1510" (UID: "49169ae5-7185-44fc-9c12-de093c4a1510"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:37:19.260635 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.260475 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-istio-data" pod "49169ae5-7185-44fc-9c12-de093c4a1510" (UID: "49169ae5-7185-44fc-9c12-de093c4a1510"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:37:19.260735 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.260712 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49169ae5-7185-44fc-9c12-de093c4a1510-istiod-ca-cert" pod "49169ae5-7185-44fc-9c12-de093c4a1510" (UID: "49169ae5-7185-44fc-9c12-de093c4a1510"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:37:19.260970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.260946 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:37:19.260970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.260970 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49169ae5-7185-44fc-9c12-de093c4a1510-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:37:19.261175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.260984 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:37:19.261175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.260996 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:37:19.261175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.261007 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:37:19.262514 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.262488 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49169ae5-7185-44fc-9c12-de093c4a1510-istio-token" pod "49169ae5-7185-44fc-9c12-de093c4a1510" (UID: "49169ae5-7185-44fc-9c12-de093c4a1510"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:37:19.262612 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.262524 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49169ae5-7185-44fc-9c12-de093c4a1510-istio-podinfo" pod "49169ae5-7185-44fc-9c12-de093c4a1510" (UID: "49169ae5-7185-44fc-9c12-de093c4a1510"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:37:19.262612 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.262558 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49169ae5-7185-44fc-9c12-de093c4a1510-kube-api-access-thbm4" pod "49169ae5-7185-44fc-9c12-de093c4a1510" (UID: "49169ae5-7185-44fc-9c12-de093c4a1510"). InnerVolumeSpecName "kube-api-access-thbm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:37:19.262703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.262688 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-istio-envoy" pod "49169ae5-7185-44fc-9c12-de093c4a1510" (UID: "49169ae5-7185-44fc-9c12-de093c4a1510"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:37:19.362639 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.362551 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49169ae5-7185-44fc-9c12-de093c4a1510-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:37:19.362639 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.362584 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49169ae5-7185-44fc-9c12-de093c4a1510-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:37:19.362639 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.362593 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thbm4\" (UniqueName: \"kubernetes.io/projected/49169ae5-7185-44fc-9c12-de093c4a1510-kube-api-access-thbm4\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:37:19.362639 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.362602 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49169ae5-7185-44fc-9c12-de093c4a1510-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:37:19.648926 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.648812 5998 generic.go:356] "Generic (PLEG): container finished" podID="49169ae5-7185-44fc-9c12-de093c4a1510" containerID="3f194d72314a0a862ae405d5bc92780190597d7b448d03584e88874a95538972" exitCode=0 Oct 30 02:37:19.648926 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.648895 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" event={"ID":"49169ae5-7185-44fc-9c12-de093c4a1510","Type":"ContainerDied","Data":"3f194d72314a0a862ae405d5bc92780190597d7b448d03584e88874a95538972"} Oct 30 02:37:19.649225 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.649016 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" event={"ID":"49169ae5-7185-44fc-9c12-de093c4a1510","Type":"ContainerDied","Data":"5845ce36782e34f2bb7d682b1b61bde6f891f4ed1388effb1b74f1ca4588bdf3"} Oct 30 02:37:19.649225 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.649074 5998 scope.go:122] "RemoveContainer" containerID="3f194d72314a0a862ae405d5bc92780190597d7b448d03584e88874a95538972" Oct 30 02:37:19.649417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.649388 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb" Oct 30 02:37:19.667463 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.667443 5998 scope.go:122] "RemoveContainer" containerID="7089b6c23ec1cc5a857889536a8e186ae6a5c177bde932a77d9f1b28205e1e0e" Oct 30 02:37:19.677670 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.677650 5998 scope.go:122] "RemoveContainer" containerID="3f194d72314a0a862ae405d5bc92780190597d7b448d03584e88874a95538972" Oct 30 02:37:19.677959 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:37:19.677937 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f194d72314a0a862ae405d5bc92780190597d7b448d03584e88874a95538972\": container with ID starting with 3f194d72314a0a862ae405d5bc92780190597d7b448d03584e88874a95538972 not found: ID does not exist" containerID="3f194d72314a0a862ae405d5bc92780190597d7b448d03584e88874a95538972" Oct 30 02:37:19.678025 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.677969 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f194d72314a0a862ae405d5bc92780190597d7b448d03584e88874a95538972"} err="failed to get container status \"3f194d72314a0a862ae405d5bc92780190597d7b448d03584e88874a95538972\": rpc error: code = NotFound desc = could not find container \"3f194d72314a0a862ae405d5bc92780190597d7b448d03584e88874a95538972\": container with ID starting with 3f194d72314a0a862ae405d5bc92780190597d7b448d03584e88874a95538972 not found: ID does not exist" Oct 30 02:37:19.678025 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.677987 5998 scope.go:122] "RemoveContainer" containerID="7089b6c23ec1cc5a857889536a8e186ae6a5c177bde932a77d9f1b28205e1e0e" Oct 30 02:37:19.678270 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:37:19.678254 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7089b6c23ec1cc5a857889536a8e186ae6a5c177bde932a77d9f1b28205e1e0e\": container with ID starting with 7089b6c23ec1cc5a857889536a8e186ae6a5c177bde932a77d9f1b28205e1e0e not found: ID does not exist" containerID="7089b6c23ec1cc5a857889536a8e186ae6a5c177bde932a77d9f1b28205e1e0e" Oct 30 02:37:19.678313 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.678274 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7089b6c23ec1cc5a857889536a8e186ae6a5c177bde932a77d9f1b28205e1e0e"} err="failed to get container status \"7089b6c23ec1cc5a857889536a8e186ae6a5c177bde932a77d9f1b28205e1e0e\": rpc error: code = NotFound desc = could not find container \"7089b6c23ec1cc5a857889536a8e186ae6a5c177bde932a77d9f1b28205e1e0e\": container with ID starting with 7089b6c23ec1cc5a857889536a8e186ae6a5c177bde932a77d9f1b28205e1e0e not found: ID does not exist" Oct 30 02:37:19.711121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.711064 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb"] Oct 30 02:37:19.725133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:19.723153 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d207e8-openshift-default-55685dccbf-v5rsb"] Oct 30 02:37:20.542931 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:20.542891 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49169ae5-7185-44fc-9c12-de093c4a1510" path="/var/lib/kubelet/pods/49169ae5-7185-44fc-9c12-de093c4a1510/volumes" Oct 30 02:37:20.681609 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:20.681571 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:22.210520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:22.210481 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:23.798241 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:23.798206 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:25.222907 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:37:25.222868 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:37:25.401937 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:25.401893 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:27.058856 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:27.058808 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:28.641958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:28.641920 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:30.206845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:30.206808 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:31.056141 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:31.056062 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:31.861720 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:31.861683 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:33.449230 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:33.449196 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:35.053543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:35.053510 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:36.687438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:36.687399 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:37.769205 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:37.769163 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:38.358146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:38.358068 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:39.513971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.513924 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf"] Oct 30 02:37:39.514870 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.514846 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7e352468-3606-467c-abd3-4654142859b0" containerName="revoke-keys" Oct 30 02:37:39.514968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.514872 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e352468-3606-467c-abd3-4654142859b0" containerName="revoke-keys" Oct 30 02:37:39.514968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.514901 5998 cpu_manager.go:414] "containerMap: removing container" podUID="49169ae5-7185-44fc-9c12-de093c4a1510" containerName="istio-proxy" Oct 30 02:37:39.514968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.514910 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49169ae5-7185-44fc-9c12-de093c4a1510" containerName="istio-proxy" Oct 30 02:37:39.514968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.514936 5998 cpu_manager.go:414] "containerMap: removing container" podUID="025cc8ef-572d-4f6a-851c-2e59fb797189" containerName="istio-proxy" Oct 30 02:37:39.514968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.514943 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025cc8ef-572d-4f6a-851c-2e59fb797189" containerName="istio-proxy" Oct 30 02:37:39.514968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.514966 5998 cpu_manager.go:414] "containerMap: removing container" podUID="2fd2791e-dcdd-40a2-be03-95c5c266d99c" containerName="istio-proxy" Oct 30 02:37:39.515332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.514974 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd2791e-dcdd-40a2-be03-95c5c266d99c" containerName="istio-proxy" Oct 30 02:37:39.515332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.514987 5998 cpu_manager.go:414] "containerMap: removing container" podUID="49169ae5-7185-44fc-9c12-de093c4a1510" containerName="istio-proxy" Oct 30 02:37:39.515332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.514996 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49169ae5-7185-44fc-9c12-de093c4a1510" containerName="istio-proxy" Oct 30 02:37:39.515332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.515013 5998 cpu_manager.go:414] "containerMap: removing container" podUID="2fd2791e-dcdd-40a2-be03-95c5c266d99c" containerName="istio-proxy" Oct 30 02:37:39.515332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.515020 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fd2791e-dcdd-40a2-be03-95c5c266d99c" containerName="istio-proxy" Oct 30 02:37:39.515332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.515197 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="025cc8ef-572d-4f6a-851c-2e59fb797189" containerName="istio-proxy" Oct 30 02:37:39.515332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.515215 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd2791e-dcdd-40a2-be03-95c5c266d99c" containerName="istio-proxy" Oct 30 02:37:39.515332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.515224 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="49169ae5-7185-44fc-9c12-de093c4a1510" containerName="istio-proxy" Oct 30 02:37:39.515332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.515240 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fd2791e-dcdd-40a2-be03-95c5c266d99c" containerName="istio-proxy" Oct 30 02:37:39.515332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.515256 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e352468-3606-467c-abd3-4654142859b0" containerName="revoke-keys" Oct 30 02:37:39.519287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.519262 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf" Oct 30 02:37:39.539684 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.539650 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf"] Oct 30 02:37:39.544264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.544238 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:37:39.544514 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.544482 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:37:39.544694 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.544489 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:37:39.544795 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.544572 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:37:39.561635 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.561601 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17631d18-zx9xf\" (UID: \"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf" Oct 30 02:37:39.561808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.561653 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ljqt\" (UniqueName: \"kubernetes.io/projected/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-kube-api-access-4ljqt\") pod \"maas-api-revoke-sub-17631d18-zx9xf\" (UID: \"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf" Oct 30 02:37:39.561894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.561845 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17631d18-zx9xf\" (UID: \"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf" Oct 30 02:37:39.663126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.663073 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17631d18-zx9xf\" (UID: \"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf" Oct 30 02:37:39.663290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.663255 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17631d18-zx9xf\" (UID: \"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf" Oct 30 02:37:39.663351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.663296 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ljqt\" (UniqueName: \"kubernetes.io/projected/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-kube-api-access-4ljqt\") pod \"maas-api-revoke-sub-17631d18-zx9xf\" (UID: \"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf" Oct 30 02:37:39.663885 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.663859 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17631d18-zx9xf\" (UID: \"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf" Oct 30 02:37:39.665401 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.665383 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17631d18-zx9xf\" (UID: \"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf" Oct 30 02:37:39.684845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.684821 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ljqt\" (UniqueName: \"kubernetes.io/projected/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-kube-api-access-4ljqt\") pod \"maas-api-revoke-sub-17631d18-zx9xf\" (UID: \"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf" Oct 30 02:37:39.836259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.836176 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf" Oct 30 02:37:39.994974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:39.994931 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:40.151187 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:40.151035 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf"] Oct 30 02:37:40.154471 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:37:40.154435 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b53f6b8_8a57_4f6b_b4fe_5a13da5a6977.slice/crio-7fe123e6273fcb2b6b8d5b67c4fffa7025c9a1dcc6cc6d22ae32a2a9573097b8 WatchSource:0}: Error finding container 7fe123e6273fcb2b6b8d5b67c4fffa7025c9a1dcc6cc6d22ae32a2a9573097b8: Status 404 returned error can't find the container with id 7fe123e6273fcb2b6b8d5b67c4fffa7025c9a1dcc6cc6d22ae32a2a9573097b8 Oct 30 02:37:40.469040 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:40.468995 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:40.828695 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:40.828653 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf" event={"ID":"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977","Type":"ContainerStarted","Data":"c953aded0e0ce0adac69aee0a8f1184b5dd6e8705fe03b6cc1a445852ea448a3"} Oct 30 02:37:40.829161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:40.828717 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf" event={"ID":"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977","Type":"ContainerStarted","Data":"7fe123e6273fcb2b6b8d5b67c4fffa7025c9a1dcc6cc6d22ae32a2a9573097b8"} Oct 30 02:37:40.869756 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:40.869698 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf" podStartSLOduration=1.869681194 podStartE2EDuration="1.869681194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:37:40.865634179 +0000 UTC m=+4090.444425260" watchObservedRunningTime="2026-10-30 02:37:40.869681194 +0000 UTC m=+4090.448472276" Oct 30 02:37:41.032831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:41.032789 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb"] Oct 30 02:37:41.036672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:41.036640 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cda67a7d-4m6hb"] Oct 30 02:37:41.493433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:41.493395 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:41.839287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:41.839192 5998 generic.go:356] "Generic (PLEG): container finished" podID="1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977" containerID="c953aded0e0ce0adac69aee0a8f1184b5dd6e8705fe03b6cc1a445852ea448a3" exitCode=0 Oct 30 02:37:41.839692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:41.839285 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf" event={"ID":"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977","Type":"ContainerDied","Data":"c953aded0e0ce0adac69aee0a8f1184b5dd6e8705fe03b6cc1a445852ea448a3"} Oct 30 02:37:42.540371 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:42.540335 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc1d4dfd-112c-410d-8f45-155b993865d7" path="/var/lib/kubelet/pods/fc1d4dfd-112c-410d-8f45-155b993865d7/volumes" Oct 30 02:37:43.011245 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:43.011211 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf" Oct 30 02:37:43.094679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:43.094643 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:43.106507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:43.106424 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-kube-api-access-4ljqt\" (UniqueName: \"kubernetes.io/projected/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-kube-api-access-4ljqt\") pod \"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977\" (UID: \"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977\") " Oct 30 02:37:43.106666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:43.106534 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-maas-api-openshift-ca\") pod \"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977\" (UID: \"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977\") " Oct 30 02:37:43.106666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:43.106587 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-maas-api-xks-ca\") pod \"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977\" (UID: \"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977\") " Oct 30 02:37:43.106920 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:43.106889 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-maas-api-openshift-ca" pod "1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977" (UID: "1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:37:43.107292 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:43.107256 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:37:43.109579 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:43.109546 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-maas-api-xks-ca" pod "1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977" (UID: "1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:37:43.109688 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:43.109665 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-kube-api-access-4ljqt" pod "1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977" (UID: "1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977"). InnerVolumeSpecName "kube-api-access-4ljqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:37:43.208571 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:43.208537 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:37:43.208571 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:43.208568 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ljqt\" (UniqueName: \"kubernetes.io/projected/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977-kube-api-access-4ljqt\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:37:43.856021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:43.855982 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf" Oct 30 02:37:43.856217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:43.855993 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf" event={"ID":"1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977","Type":"ContainerDied","Data":"7fe123e6273fcb2b6b8d5b67c4fffa7025c9a1dcc6cc6d22ae32a2a9573097b8"} Oct 30 02:37:43.856217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:43.856164 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fe123e6273fcb2b6b8d5b67c4fffa7025c9a1dcc6cc6d22ae32a2a9573097b8" Oct 30 02:37:44.675252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:44.675207 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:46.169820 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:46.169788 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:47.886273 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:47.886237 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:47.942417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:47.942328 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:37:48.669602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:48.669559 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:49.140190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.140082 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx"] Oct 30 02:37:49.141062 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.141025 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977" containerName="revoke-keys" Oct 30 02:37:49.141062 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.141054 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977" containerName="revoke-keys" Oct 30 02:37:49.141320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.141237 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977" containerName="revoke-keys" Oct 30 02:37:49.141320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.141253 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="49169ae5-7185-44fc-9c12-de093c4a1510" containerName="istio-proxy" Oct 30 02:37:49.145838 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.145813 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.167861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.167832 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-edae4317-openshift-default-dockercfg-kqlz7\"" Oct 30 02:37:49.184333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.184283 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx"] Oct 30 02:37:49.194898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.194863 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-token\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.195069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.194905 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-envoy\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.195069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.195028 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-workload-certs\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.195322 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.195295 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-workload-socket\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.195517 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.195473 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-credential-socket\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.195616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.195560 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-podinfo\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.195616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.195602 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-data\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.195747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.195681 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7460a2-c42d-4d5f-964e-525a49cdafff-istiod-ca-cert\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.195747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.195707 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfs27\" (UniqueName: \"kubernetes.io/projected/dd7460a2-c42d-4d5f-964e-525a49cdafff-kube-api-access-rfs27\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.276462 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.276418 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:49.297034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.296986 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-workload-socket\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.297277 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.297059 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-credential-socket\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.297277 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.297144 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-podinfo\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.297277 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.297184 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-data\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.297491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.297289 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7460a2-c42d-4d5f-964e-525a49cdafff-istiod-ca-cert\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.297491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.297317 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfs27\" (UniqueName: \"kubernetes.io/projected/dd7460a2-c42d-4d5f-964e-525a49cdafff-kube-api-access-rfs27\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.297491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.297360 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-token\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.297491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.297393 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-envoy\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.297491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.297457 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-workload-certs\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.300033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.300000 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7460a2-c42d-4d5f-964e-525a49cdafff-istiod-ca-cert\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.300695 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.300667 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-workload-socket\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.301875 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.301846 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-credential-socket\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.315262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.315219 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-data\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.315835 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.315807 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-workload-certs\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.338799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.338764 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-podinfo\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.380944 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.380446 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-envoy\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.384122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.381794 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfs27\" (UniqueName: \"kubernetes.io/projected/dd7460a2-c42d-4d5f-964e-525a49cdafff-kube-api-access-rfs27\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.384122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.382307 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-token\") pod \"e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.400193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.399963 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:49.464436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.464384 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:49.655618 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.655538 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:49.933564 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.933535 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx"] Oct 30 02:37:49.936050 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:37:49.936012 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7460a2_c42d_4d5f_964e_525a49cdafff.slice/crio-d88d545e00226a0380765c959edf5b08069888eaf469146c695a882f352348f9 WatchSource:0}: Error finding container d88d545e00226a0380765c959edf5b08069888eaf469146c695a882f352348f9: Status 404 returned error can't find the container with id d88d545e00226a0380765c959edf5b08069888eaf469146c695a882f352348f9 Oct 30 02:37:49.939561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.939522 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:37:49.939677 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.939637 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:37:49.939744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:49.939682 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:37:50.616078 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:50.616034 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv"] Oct 30 02:37:50.679117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:50.669349 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:50.914735 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:50.914695 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" event={"ID":"dd7460a2-c42d-4d5f-964e-525a49cdafff","Type":"ContainerStarted","Data":"c0a1c55f47d781e132faae60ef728cf9a1cbff25fc1a3a59920398e2394e9579"} Oct 30 02:37:50.914919 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:50.914766 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" event={"ID":"dd7460a2-c42d-4d5f-964e-525a49cdafff","Type":"ContainerStarted","Data":"d88d545e00226a0380765c959edf5b08069888eaf469146c695a882f352348f9"} Oct 30 02:37:50.960766 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:50.960704 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" podStartSLOduration=1.960683946 podStartE2EDuration="1.960683946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:37:50.959275671 +0000 UTC m=+4100.538066754" watchObservedRunningTime="2026-10-30 02:37:50.960683946 +0000 UTC m=+4100.539475028" Oct 30 02:37:51.137385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.137348 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s"] Oct 30 02:37:51.154453 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.154419 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s" Oct 30 02:37:51.162628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.162567 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s"] Oct 30 02:37:51.170263 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.169215 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:37:51.170263 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.169942 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:37:51.170263 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.170077 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:37:51.182286 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.182249 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:37:51.236510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.236462 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd980138-lww2s\" (UID: \"7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s" Oct 30 02:37:51.236744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.236607 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd980138-lww2s\" (UID: \"7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s" Oct 30 02:37:51.236744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.236647 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kprfj\" (UniqueName: \"kubernetes.io/projected/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-kube-api-access-kprfj\") pod \"maas-api-revoke-sub-cd980138-lww2s\" (UID: \"7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s" Oct 30 02:37:51.338798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.338750 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd980138-lww2s\" (UID: \"7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s" Oct 30 02:37:51.338998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.338910 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd980138-lww2s\" (UID: \"7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s" Oct 30 02:37:51.338998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.338953 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kprfj\" (UniqueName: \"kubernetes.io/projected/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-kube-api-access-kprfj\") pod \"maas-api-revoke-sub-cd980138-lww2s\" (UID: \"7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s" Oct 30 02:37:51.339587 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.339548 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd980138-lww2s\" (UID: \"7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s" Oct 30 02:37:51.341346 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.341322 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd980138-lww2s\" (UID: \"7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s" Oct 30 02:37:51.361023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.360989 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kprfj\" (UniqueName: \"kubernetes.io/projected/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-kube-api-access-kprfj\") pod \"maas-api-revoke-sub-cd980138-lww2s\" (UID: \"7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s" Oct 30 02:37:51.466509 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.466414 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:51.472900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.472869 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:51.481816 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.481786 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s" Oct 30 02:37:51.721252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.718664 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:51.806321 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.806290 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s"] Oct 30 02:37:51.809549 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:37:51.809510 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af8fdff_1df6_4cf2_9cc1_4a5306c9a14c.slice/crio-8de41fd102f13ad684b35dcce48399d73def583490071fe2b39228462e0bb8c4 WatchSource:0}: Error finding container 8de41fd102f13ad684b35dcce48399d73def583490071fe2b39228462e0bb8c4: Status 404 returned error can't find the container with id 8de41fd102f13ad684b35dcce48399d73def583490071fe2b39228462e0bb8c4 Oct 30 02:37:51.925162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.925080 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s" event={"ID":"7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c","Type":"ContainerStarted","Data":"8de41fd102f13ad684b35dcce48399d73def583490071fe2b39228462e0bb8c4"} Oct 30 02:37:51.925542 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.925520 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:51.926527 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:51.926501 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:37:52.356681 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:52.356572 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:52.477346 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:52.477306 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:52.936417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:52.936380 5998 generic.go:356] "Generic (PLEG): container finished" podID="7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c" containerID="d52f8bfbdb6fc2296bafc9735a43b71b2e5147294c78ccf65a5f412eb4396df2" exitCode=0 Oct 30 02:37:52.936869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:52.936472 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s" event={"ID":"7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c","Type":"ContainerDied","Data":"d52f8bfbdb6fc2296bafc9735a43b71b2e5147294c78ccf65a5f412eb4396df2"} Oct 30 02:37:53.285623 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:53.285584 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:53.657146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:53.657032 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:53.795901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:53.795867 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:54.111235 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:54.111200 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s" Oct 30 02:37:54.180490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:54.180456 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-maas-api-openshift-ca\") pod \"7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c\" (UID: \"7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c\") " Oct 30 02:37:54.180688 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:54.180511 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-maas-api-xks-ca\") pod \"7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c\" (UID: \"7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c\") " Oct 30 02:37:54.180688 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:54.180624 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-kube-api-access-kprfj\" (UniqueName: \"kubernetes.io/projected/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-kube-api-access-kprfj\") pod \"7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c\" (UID: \"7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c\") " Oct 30 02:37:54.180905 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:54.180880 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-maas-api-openshift-ca" pod "7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c" (UID: "7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:37:54.181492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:54.181460 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:37:54.183028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:54.183004 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-maas-api-xks-ca" pod "7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c" (UID: "7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:37:54.183451 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:54.183429 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-kube-api-access-kprfj" pod "7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c" (UID: "7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c"). InnerVolumeSpecName "kube-api-access-kprfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:37:54.283056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:54.282969 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:37:54.283056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:54.283003 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kprfj\" (UniqueName: \"kubernetes.io/projected/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c-kube-api-access-kprfj\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:37:54.953818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:54.953786 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s" Oct 30 02:37:54.954004 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:54.953779 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s" event={"ID":"7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c","Type":"ContainerDied","Data":"8de41fd102f13ad684b35dcce48399d73def583490071fe2b39228462e0bb8c4"} Oct 30 02:37:54.954004 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:54.953925 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8de41fd102f13ad684b35dcce48399d73def583490071fe2b39228462e0bb8c4" Oct 30 02:37:55.175298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.175258 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:55.234800 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:37:55.234705 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:37:55.289099 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.289065 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:55.372209 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.372171 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z"] Oct 30 02:37:55.372803 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.372786 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c" containerName="revoke-keys" Oct 30 02:37:55.372853 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.372803 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c" containerName="revoke-keys" Oct 30 02:37:55.372899 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.372890 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c" containerName="revoke-keys" Oct 30 02:37:55.376775 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.376750 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z" Oct 30 02:37:55.389675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.389640 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:37:55.399900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.399862 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z"] Oct 30 02:37:55.496174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.488463 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:37:55.496174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.488668 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:37:55.496174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.488717 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:37:55.527142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.523117 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:55.533124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.528589 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg5rt\" (UniqueName: \"kubernetes.io/projected/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-kube-api-access-xg5rt\") pod \"maas-api-revoke-sub-dbb9d0b5-6gp8z\" (UID: \"30eb3269-4d9d-4423-ab4d-0b4ff17d2169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z" Oct 30 02:37:55.533124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.528652 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbb9d0b5-6gp8z\" (UID: \"30eb3269-4d9d-4423-ab4d-0b4ff17d2169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z" Oct 30 02:37:55.533124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.528817 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbb9d0b5-6gp8z\" (UID: \"30eb3269-4d9d-4423-ab4d-0b4ff17d2169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z" Oct 30 02:37:55.638126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.637044 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg5rt\" (UniqueName: \"kubernetes.io/projected/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-kube-api-access-xg5rt\") pod \"maas-api-revoke-sub-dbb9d0b5-6gp8z\" (UID: \"30eb3269-4d9d-4423-ab4d-0b4ff17d2169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z" Oct 30 02:37:55.638126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.637119 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbb9d0b5-6gp8z\" (UID: \"30eb3269-4d9d-4423-ab4d-0b4ff17d2169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z" Oct 30 02:37:55.638126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.637324 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbb9d0b5-6gp8z\" (UID: \"30eb3269-4d9d-4423-ab4d-0b4ff17d2169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z" Oct 30 02:37:55.649140 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.644226 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbb9d0b5-6gp8z\" (UID: \"30eb3269-4d9d-4423-ab4d-0b4ff17d2169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z" Oct 30 02:37:55.727661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.727499 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbb9d0b5-6gp8z\" (UID: \"30eb3269-4d9d-4423-ab4d-0b4ff17d2169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z" Oct 30 02:37:55.767229 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.767030 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg5rt\" (UniqueName: \"kubernetes.io/projected/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-kube-api-access-xg5rt\") pod \"maas-api-revoke-sub-dbb9d0b5-6gp8z\" (UID: \"30eb3269-4d9d-4423-ab4d-0b4ff17d2169\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z" Oct 30 02:37:55.992167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:55.992133 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z" Oct 30 02:37:56.306017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:56.305984 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z"] Oct 30 02:37:56.308725 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:37:56.308654 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30eb3269_4d9d_4423_ab4d_0b4ff17d2169.slice/crio-7fddc2d2265c459c15d5ca257115df6066421f89fc5fc8f043278cca6b5314dc WatchSource:0}: Error finding container 7fddc2d2265c459c15d5ca257115df6066421f89fc5fc8f043278cca6b5314dc: Status 404 returned error can't find the container with id 7fddc2d2265c459c15d5ca257115df6066421f89fc5fc8f043278cca6b5314dc Oct 30 02:37:56.970429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:56.970320 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z" event={"ID":"30eb3269-4d9d-4423-ab4d-0b4ff17d2169","Type":"ContainerStarted","Data":"d3b641a80ce11c270a6c07b6217c02580315a1ae126b1d23b59e054e3a962d81"} Oct 30 02:37:56.970429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:56.970400 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z" event={"ID":"30eb3269-4d9d-4423-ab4d-0b4ff17d2169","Type":"ContainerStarted","Data":"7fddc2d2265c459c15d5ca257115df6066421f89fc5fc8f043278cca6b5314dc"} Oct 30 02:37:57.012699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:57.012636 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z" podStartSLOduration=2.012619751 podStartE2EDuration="2.012619751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:37:57.007709987 +0000 UTC m=+4106.586501070" watchObservedRunningTime="2026-10-30 02:37:57.012619751 +0000 UTC m=+4106.591410831" Oct 30 02:37:57.164614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:57.164574 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:57.979774 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:57.979735 5998 generic.go:356] "Generic (PLEG): container finished" podID="30eb3269-4d9d-4423-ab4d-0b4ff17d2169" containerID="d3b641a80ce11c270a6c07b6217c02580315a1ae126b1d23b59e054e3a962d81" exitCode=0 Oct 30 02:37:57.980255 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:57.979821 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z" event={"ID":"30eb3269-4d9d-4423-ab4d-0b4ff17d2169","Type":"ContainerDied","Data":"d3b641a80ce11c270a6c07b6217c02580315a1ae126b1d23b59e054e3a962d81"} Oct 30 02:37:58.723762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:58.723717 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:37:59.180535 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:59.180509 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z" Oct 30 02:37:59.330698 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:59.330596 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-maas-api-openshift-ca\") pod \"30eb3269-4d9d-4423-ab4d-0b4ff17d2169\" (UID: \"30eb3269-4d9d-4423-ab4d-0b4ff17d2169\") " Oct 30 02:37:59.330882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:59.330740 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-maas-api-xks-ca\") pod \"30eb3269-4d9d-4423-ab4d-0b4ff17d2169\" (UID: \"30eb3269-4d9d-4423-ab4d-0b4ff17d2169\") " Oct 30 02:37:59.330882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:59.330834 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-kube-api-access-xg5rt\" (UniqueName: \"kubernetes.io/projected/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-kube-api-access-xg5rt\") pod \"30eb3269-4d9d-4423-ab4d-0b4ff17d2169\" (UID: \"30eb3269-4d9d-4423-ab4d-0b4ff17d2169\") " Oct 30 02:37:59.331052 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:59.331022 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-maas-api-openshift-ca" pod "30eb3269-4d9d-4423-ab4d-0b4ff17d2169" (UID: "30eb3269-4d9d-4423-ab4d-0b4ff17d2169"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:37:59.332170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:59.332144 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:37:59.335484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:59.335454 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-maas-api-xks-ca" pod "30eb3269-4d9d-4423-ab4d-0b4ff17d2169" (UID: "30eb3269-4d9d-4423-ab4d-0b4ff17d2169"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:37:59.335746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:59.335723 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-kube-api-access-xg5rt" pod "30eb3269-4d9d-4423-ab4d-0b4ff17d2169" (UID: "30eb3269-4d9d-4423-ab4d-0b4ff17d2169"). InnerVolumeSpecName "kube-api-access-xg5rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:37:59.434052 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:59.434007 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:37:59.434052 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:59.434053 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg5rt\" (UniqueName: \"kubernetes.io/projected/30eb3269-4d9d-4423-ab4d-0b4ff17d2169-kube-api-access-xg5rt\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:37:59.998263 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:59.998230 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z" Oct 30 02:37:59.998465 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:59.998275 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z" event={"ID":"30eb3269-4d9d-4423-ab4d-0b4ff17d2169","Type":"ContainerDied","Data":"7fddc2d2265c459c15d5ca257115df6066421f89fc5fc8f043278cca6b5314dc"} Oct 30 02:37:59.998465 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:37:59.998334 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fddc2d2265c459c15d5ca257115df6066421f89fc5fc8f043278cca6b5314dc" Oct 30 02:38:00.289071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:00.288981 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:01.862650 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:01.862612 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:02.034996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:02.034949 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz"] Oct 30 02:38:02.042306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:02.042273 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a8b10b9-ttrtz"] Oct 30 02:38:02.543275 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:02.543238 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e33e5cb-5be4-4ac8-ab50-506cac62ae3f" path="/var/lib/kubelet/pods/7e33e5cb-5be4-4ac8-ab50-506cac62ae3f/volumes" Oct 30 02:38:03.475361 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:03.475320 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:05.111317 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:05.111280 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:05.645231 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:05.645184 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" podUID="76604767-dec2-42cd-b953-65221160999f" containerName="main" containerID="cri-o://d120ee65e7cb7507ccde5a41227f595de7ea1305580edc4c52527fd1b6218e7f" gracePeriod=45 Oct 30 02:38:05.836432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:05.824678 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:05.990318 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:05.989905 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" podUID="76604767-dec2-42cd-b953-65221160999f" containerName="main" probeResult="failure" output="Get \"https://10.133.1.28:8000/ready\": dial tcp 10.133.1.28:8000: connect: connection refused" Oct 30 02:38:06.092356 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.092316 5998 generic.go:356] "Generic (PLEG): container finished" podID="76604767-dec2-42cd-b953-65221160999f" containerID="d120ee65e7cb7507ccde5a41227f595de7ea1305580edc4c52527fd1b6218e7f" exitCode=0 Oct 30 02:38:06.092565 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.092390 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" event={"ID":"76604767-dec2-42cd-b953-65221160999f","Type":"ContainerDied","Data":"d120ee65e7cb7507ccde5a41227f595de7ea1305580edc4c52527fd1b6218e7f"} Oct 30 02:38:06.133746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.133719 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:38:06.164766 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.164724 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-dshm\") pod \"76604767-dec2-42cd-b953-65221160999f\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " Oct 30 02:38:06.164968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.164836 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76604767-dec2-42cd-b953-65221160999f-kube-api-access-2zt8g\" (UniqueName: \"kubernetes.io/projected/76604767-dec2-42cd-b953-65221160999f-kube-api-access-2zt8g\") pod \"76604767-dec2-42cd-b953-65221160999f\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " Oct 30 02:38:06.165035 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.164996 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-model-cache\") pod \"76604767-dec2-42cd-b953-65221160999f\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " Oct 30 02:38:06.165135 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.165033 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76604767-dec2-42cd-b953-65221160999f-tls-certs\" (UniqueName: \"kubernetes.io/secret/76604767-dec2-42cd-b953-65221160999f-tls-certs\") pod \"76604767-dec2-42cd-b953-65221160999f\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " Oct 30 02:38:06.165135 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.165081 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-tmp-dir\") pod \"76604767-dec2-42cd-b953-65221160999f\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " Oct 30 02:38:06.165276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.165198 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-home\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-home\") pod \"76604767-dec2-42cd-b953-65221160999f\" (UID: \"76604767-dec2-42cd-b953-65221160999f\") " Oct 30 02:38:06.165356 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.165323 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-model-cache" pod "76604767-dec2-42cd-b953-65221160999f" (UID: "76604767-dec2-42cd-b953-65221160999f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:38:06.165429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.165394 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-tmp-dir" pod "76604767-dec2-42cd-b953-65221160999f" (UID: "76604767-dec2-42cd-b953-65221160999f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:38:06.165557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.165532 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-home" pod "76604767-dec2-42cd-b953-65221160999f" (UID: "76604767-dec2-42cd-b953-65221160999f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:38:06.165937 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.165914 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:06.165988 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.165969 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:06.165988 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.165985 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:06.167307 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.167284 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-dshm" pod "76604767-dec2-42cd-b953-65221160999f" (UID: "76604767-dec2-42cd-b953-65221160999f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:38:06.167545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.167522 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76604767-dec2-42cd-b953-65221160999f-tls-certs" pod "76604767-dec2-42cd-b953-65221160999f" (UID: "76604767-dec2-42cd-b953-65221160999f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:38:06.167642 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.167530 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76604767-dec2-42cd-b953-65221160999f-kube-api-access-2zt8g" pod "76604767-dec2-42cd-b953-65221160999f" (UID: "76604767-dec2-42cd-b953-65221160999f"). InnerVolumeSpecName "kube-api-access-2zt8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:38:06.267040 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.266950 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76604767-dec2-42cd-b953-65221160999f-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:06.267040 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.266983 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zt8g\" (UniqueName: \"kubernetes.io/projected/76604767-dec2-42cd-b953-65221160999f-kube-api-access-2zt8g\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:06.267040 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.266993 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76604767-dec2-42cd-b953-65221160999f-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:06.769068 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:06.769036 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:07.103121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:07.103071 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" Oct 30 02:38:07.103416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:07.103081 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv" event={"ID":"76604767-dec2-42cd-b953-65221160999f","Type":"ContainerDied","Data":"36bdd021c4ae7ece82de99c4981ab0a03dcc4c248a0cd3563544f72a932df07c"} Oct 30 02:38:07.103567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:07.103440 5998 scope.go:122] "RemoveContainer" containerID="d120ee65e7cb7507ccde5a41227f595de7ea1305580edc4c52527fd1b6218e7f" Oct 30 02:38:07.151282 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:07.151245 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv"] Oct 30 02:38:07.161354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:07.161324 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-52d131/test-model-52d131-kserve-644fd5978c-c6stv"] Oct 30 02:38:08.367042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:08.367005 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:08.541115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:08.541067 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76604767-dec2-42cd-b953-65221160999f" path="/var/lib/kubelet/pods/76604767-dec2-42cd-b953-65221160999f/volumes" Oct 30 02:38:09.966860 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:09.966823 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:10.260641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:10.260536 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7"] Oct 30 02:38:10.291320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:10.291289 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:10.735517 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:10.735484 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:11.577709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:11.577663 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:12.663251 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:12.663216 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:13.256791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:13.256750 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:14.794334 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:14.794296 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:16.378530 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:16.378494 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:17.596524 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:17.596482 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:17.703934 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:17.703893 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:17.810369 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:17.810327 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:18.034208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:18.034170 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:19.682175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:19.682144 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:21.190397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:21.190354 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:22.877425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:22.877393 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:24.221301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:24.221263 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:24.254839 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:24.254804 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:38:24.666172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:24.664016 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:24.991729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:24.991645 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:25.220880 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:38:25.220845 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:38:25.323465 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:25.323362 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" podUID="9ced29b5-0313-45a9-8a67-54e28ebc4bf2" containerName="main" containerID="cri-o://5329e1e78cd2ea5551454685834db8b69319b552d21094794da3798403fee788" gracePeriod=45 Oct 30 02:38:25.760758 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:25.760333 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:38:25.846490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:25.846454 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-model-cache\") pod \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " Oct 30 02:38:25.872295 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:25.872256 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-home\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-home\") pod \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " Oct 30 02:38:25.880397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:25.880362 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-tls-certs\" (UniqueName: \"kubernetes.io/secret/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-tls-certs\") pod \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " Oct 30 02:38:25.887147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:25.887108 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-dshm\") pod \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " Oct 30 02:38:25.895641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:25.861246 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-model-cache" pod "9ced29b5-0313-45a9-8a67-54e28ebc4bf2" (UID: "9ced29b5-0313-45a9-8a67-54e28ebc4bf2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:38:25.901116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:25.880214 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-home" pod "9ced29b5-0313-45a9-8a67-54e28ebc4bf2" (UID: "9ced29b5-0313-45a9-8a67-54e28ebc4bf2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:38:25.901537 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:25.896167 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-tmp-dir" pod "9ced29b5-0313-45a9-8a67-54e28ebc4bf2" (UID: "9ced29b5-0313-45a9-8a67-54e28ebc4bf2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:38:25.901727 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:25.901455 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-dshm" pod "9ced29b5-0313-45a9-8a67-54e28ebc4bf2" (UID: "9ced29b5-0313-45a9-8a67-54e28ebc4bf2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:38:25.930206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:25.924165 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-tmp-dir\") pod \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " Oct 30 02:38:25.930206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:25.924357 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-kube-api-access-2kkgx\" (UniqueName: \"kubernetes.io/projected/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-kube-api-access-2kkgx\") pod \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\" (UID: \"9ced29b5-0313-45a9-8a67-54e28ebc4bf2\") " Oct 30 02:38:25.930206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:25.925047 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:25.930206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:25.925068 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:25.930206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:25.925081 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:25.930206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:25.925113 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:25.941842 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:25.941799 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-kube-api-access-2kkgx" pod "9ced29b5-0313-45a9-8a67-54e28ebc4bf2" (UID: "9ced29b5-0313-45a9-8a67-54e28ebc4bf2"). InnerVolumeSpecName "kube-api-access-2kkgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:38:25.943894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:25.943854 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-tls-certs" pod "9ced29b5-0313-45a9-8a67-54e28ebc4bf2" (UID: "9ced29b5-0313-45a9-8a67-54e28ebc4bf2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:38:26.026264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:26.026228 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kkgx\" (UniqueName: \"kubernetes.io/projected/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-kube-api-access-2kkgx\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:26.026264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:26.026262 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ced29b5-0313-45a9-8a67-54e28ebc4bf2-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:26.450854 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:26.450811 5998 generic.go:356] "Generic (PLEG): container finished" podID="9ced29b5-0313-45a9-8a67-54e28ebc4bf2" containerID="5329e1e78cd2ea5551454685834db8b69319b552d21094794da3798403fee788" exitCode=0 Oct 30 02:38:26.451312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:26.450875 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" event={"ID":"9ced29b5-0313-45a9-8a67-54e28ebc4bf2","Type":"ContainerDied","Data":"5329e1e78cd2ea5551454685834db8b69319b552d21094794da3798403fee788"} Oct 30 02:38:26.451312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:26.450917 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" event={"ID":"9ced29b5-0313-45a9-8a67-54e28ebc4bf2","Type":"ContainerDied","Data":"5f6d16875a8f07be1bfce3ebcd8c0c7029de96e92fbc5e62602acc2cedc5abff"} Oct 30 02:38:26.451312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:26.450942 5998 scope.go:122] "RemoveContainer" containerID="5329e1e78cd2ea5551454685834db8b69319b552d21094794da3798403fee788" Oct 30 02:38:26.451312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:26.451132 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7" Oct 30 02:38:26.465397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:26.465359 5998 scope.go:122] "RemoveContainer" containerID="5329e1e78cd2ea5551454685834db8b69319b552d21094794da3798403fee788" Oct 30 02:38:26.465791 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:38:26.465757 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5329e1e78cd2ea5551454685834db8b69319b552d21094794da3798403fee788\": container with ID starting with 5329e1e78cd2ea5551454685834db8b69319b552d21094794da3798403fee788 not found: ID does not exist" containerID="5329e1e78cd2ea5551454685834db8b69319b552d21094794da3798403fee788" Oct 30 02:38:26.465957 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:26.465799 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5329e1e78cd2ea5551454685834db8b69319b552d21094794da3798403fee788"} err="failed to get container status \"5329e1e78cd2ea5551454685834db8b69319b552d21094794da3798403fee788\": rpc error: code = NotFound desc = could not find container \"5329e1e78cd2ea5551454685834db8b69319b552d21094794da3798403fee788\": container with ID starting with 5329e1e78cd2ea5551454685834db8b69319b552d21094794da3798403fee788 not found: ID does not exist" Oct 30 02:38:26.509454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:26.509404 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7"] Oct 30 02:38:26.513360 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:26.513332 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-07fb25/test-model-07fb25-kserve-646d9d44cf-qx8p7"] Oct 30 02:38:26.545188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:26.545144 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ced29b5-0313-45a9-8a67-54e28ebc4bf2" path="/var/lib/kubelet/pods/9ced29b5-0313-45a9-8a67-54e28ebc4bf2/volumes" Oct 30 02:38:26.664907 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:26.664869 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:28.449228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:28.449177 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:38:28.458123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:28.450326 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:28.622410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:28.622334 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:38:28.924135 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:28.916187 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:38:29.189168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:29.181203 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 02:38:29.214115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:29.210233 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 02:38:29.214463 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:29.214425 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 02:38:29.357115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:29.354889 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:38:29.428118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:29.407233 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:38:29.469172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:29.467229 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded" Oct 30 02:38:29.469172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:29.467923 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 02:38:29.504115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:29.500422 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 02:38:29.628182 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:29.628133 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 02:38:29.645116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:29.637015 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 02:38:29.677734 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:29.677704 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 02:38:30.293753 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:30.293714 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:31.298626 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:31.298545 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:31.526434 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:31.526335 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:32.015046 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:32.014996 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:32.120610 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:32.120570 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:32.410524 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:32.410493 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j"] Oct 30 02:38:32.411009 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:32.410785 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" podUID="cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" containerName="istio-proxy" containerID="cri-o://953e0b47c15e8a44542b8d2abafc674f8aa8204d722fce31498c13b239e8f3ca" gracePeriod=30 Oct 30 02:38:32.440696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:32.440654 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:33.226547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:33.226505 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:33.782758 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:33.782725 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:35.351413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:35.351379 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:36.832553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:36.832516 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:37.120615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.120505 5998 patch_prober.go:29] interesting pod/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:38:37.120615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.120595 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" podUID="cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:38:37.698565 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.698529 5998 generic.go:356] "Generic (PLEG): container finished" podID="cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" containerID="953e0b47c15e8a44542b8d2abafc674f8aa8204d722fce31498c13b239e8f3ca" exitCode=0 Oct 30 02:38:37.698721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.698595 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" event={"ID":"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c","Type":"ContainerDied","Data":"953e0b47c15e8a44542b8d2abafc674f8aa8204d722fce31498c13b239e8f3ca"} Oct 30 02:38:37.698721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.698678 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" event={"ID":"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c","Type":"ContainerDied","Data":"b909365d4a3c22529b9779a609a31c68dc12554c0e0ccad214c702cf1d1332fc"} Oct 30 02:38:37.698721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.698697 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b909365d4a3c22529b9779a609a31c68dc12554c0e0ccad214c702cf1d1332fc" Oct 30 02:38:37.698721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.698714 5998 scope.go:122] "RemoveContainer" containerID="c1eaa521288148421bc1583339bf04077523312cebfc7375f7db69d923bdcfa5" Oct 30 02:38:37.702007 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.701981 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:38:37.828102 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.828035 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-token\" (UniqueName: \"kubernetes.io/projected/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-token\") pod \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " Oct 30 02:38:37.828316 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.828147 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-kube-api-access-9j5gs\" (UniqueName: \"kubernetes.io/projected/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-kube-api-access-9j5gs\") pod \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " Oct 30 02:38:37.828316 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.828264 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-workload-socket\") pod \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " Oct 30 02:38:37.828451 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.828315 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istiod-ca-cert\") pod \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " Oct 30 02:38:37.828451 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.828364 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-credential-socket\") pod \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " Oct 30 02:38:37.828451 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.828398 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-workload-certs\") pod \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " Oct 30 02:38:37.828603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.828485 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-podinfo\") pod \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " Oct 30 02:38:37.828603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.828570 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-envoy\") pod \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " Oct 30 02:38:37.828720 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.828650 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-data\") pod \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\" (UID: \"cea4d2cb-cdaa-4ce4-abee-c2866af53c2c\") " Oct 30 02:38:37.828940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.828839 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-workload-certs" pod "cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" (UID: "cea4d2cb-cdaa-4ce4-abee-c2866af53c2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:38:37.828940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.828849 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istiod-ca-cert" pod "cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" (UID: "cea4d2cb-cdaa-4ce4-abee-c2866af53c2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:38:37.829710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.829013 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-credential-socket" pod "cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" (UID: "cea4d2cb-cdaa-4ce4-abee-c2866af53c2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:38:37.829710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.829200 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-data" pod "cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" (UID: "cea4d2cb-cdaa-4ce4-abee-c2866af53c2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:38:37.829710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.829352 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-workload-socket" pod "cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" (UID: "cea4d2cb-cdaa-4ce4-abee-c2866af53c2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:38:37.829710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.829448 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:37.829710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.829470 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:37.829710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.829485 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:37.829710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.829497 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:37.830941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.830918 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-kube-api-access-9j5gs" pod "cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" (UID: "cea4d2cb-cdaa-4ce4-abee-c2866af53c2c"). InnerVolumeSpecName "kube-api-access-9j5gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:38:37.831050 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.830952 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-token" pod "cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" (UID: "cea4d2cb-cdaa-4ce4-abee-c2866af53c2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:38:37.831128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.831074 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-envoy" pod "cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" (UID: "cea4d2cb-cdaa-4ce4-abee-c2866af53c2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:38:37.832412 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.832388 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-podinfo" pod "cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" (UID: "cea4d2cb-cdaa-4ce4-abee-c2866af53c2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:38:37.931191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.931078 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:37.931191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.931136 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:37.931191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.931146 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:37.931191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.931155 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:37.931191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:37.931164 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j5gs\" (UniqueName: \"kubernetes.io/projected/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c-kube-api-access-9j5gs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:38:38.451839 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:38.451805 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:38.582236 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:38.582194 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:38.707337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:38.707261 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j" Oct 30 02:38:38.774678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:38.774646 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:38.801392 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:38.801357 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j"] Oct 30 02:38:38.811348 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:38.811314 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-52d131-openshift-default-67cf47669f-pvz6j"] Oct 30 02:38:38.898103 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:38.898055 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:39.109053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:39.108966 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:39.265564 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:39.265513 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:39.441913 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:39.441854 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:40.477415 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:40.477349 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:40.573270 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:40.573232 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" path="/var/lib/kubelet/pods/cea4d2cb-cdaa-4ce4-abee-c2866af53c2c/volumes" Oct 30 02:38:40.595508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:40.595474 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:41.971514 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:41.971425 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:42.098402 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:42.098358 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:43.657081 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:43.657035 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:45.273423 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:45.273390 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:46.448798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:46.448753 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:46.553442 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:46.553369 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:46.881894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:46.881806 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:47.977750 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:47.977710 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r"] Oct 30 02:38:47.978717 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:47.978690 5998 cpu_manager.go:414] "containerMap: removing container" podUID="cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" containerName="istio-proxy" Oct 30 02:38:47.978717 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:47.978718 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" containerName="istio-proxy" Oct 30 02:38:47.978925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:47.978763 5998 cpu_manager.go:414] "containerMap: removing container" podUID="cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" containerName="istio-proxy" Oct 30 02:38:47.978925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:47.978772 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" containerName="istio-proxy" Oct 30 02:38:47.978925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:47.978816 5998 cpu_manager.go:414] "containerMap: removing container" podUID="9ced29b5-0313-45a9-8a67-54e28ebc4bf2" containerName="main" Oct 30 02:38:47.978925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:47.978825 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ced29b5-0313-45a9-8a67-54e28ebc4bf2" containerName="main" Oct 30 02:38:47.978925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:47.978836 5998 cpu_manager.go:414] "containerMap: removing container" podUID="30eb3269-4d9d-4423-ab4d-0b4ff17d2169" containerName="revoke-keys" Oct 30 02:38:47.978925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:47.978844 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30eb3269-4d9d-4423-ab4d-0b4ff17d2169" containerName="revoke-keys" Oct 30 02:38:47.978925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:47.978875 5998 cpu_manager.go:414] "containerMap: removing container" podUID="76604767-dec2-42cd-b953-65221160999f" containerName="main" Oct 30 02:38:47.978925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:47.978882 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76604767-dec2-42cd-b953-65221160999f" containerName="main" Oct 30 02:38:47.979425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:47.979023 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ced29b5-0313-45a9-8a67-54e28ebc4bf2" containerName="main" Oct 30 02:38:47.979425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:47.979035 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="30eb3269-4d9d-4423-ab4d-0b4ff17d2169" containerName="revoke-keys" Oct 30 02:38:47.979425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:47.979049 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="76604767-dec2-42cd-b953-65221160999f" containerName="main" Oct 30 02:38:47.979425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:47.979064 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" containerName="istio-proxy" Oct 30 02:38:47.979425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:47.979075 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea4d2cb-cdaa-4ce4-abee-c2866af53c2c" containerName="istio-proxy" Oct 30 02:38:47.983517 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:47.983492 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.003597 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.003565 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r"] Oct 30 02:38:48.008811 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.008780 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-2819d2\"/\"openshift-service-ca.crt\"" Oct 30 02:38:48.008811 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.008804 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-2819d2\"/\"kube-root-ca.crt\"" Oct 30 02:38:48.013604 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.013571 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-2819d2\"/\"default-dockercfg-kjwph\"" Oct 30 02:38:48.013809 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.013650 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-2819d2\"/\"sub-test-model-2819d2-kserve-self-signed-certs\"" Oct 30 02:38:48.024896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.024860 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-tmp-dir\") pod \"sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.025136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.024966 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-dshm\") pod \"sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.025136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.024998 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-tls-certs\") pod \"sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.025136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.025082 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-home\") pod \"sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.025431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.025136 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-model-cache\") pod \"sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.025431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.025235 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnzh2\" (UniqueName: \"kubernetes.io/projected/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-kube-api-access-jnzh2\") pod \"sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.030213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.030173 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:48.126531 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.126483 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-tmp-dir\") pod \"sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.126741 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.126591 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-dshm\") pod \"sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.126741 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.126631 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-tls-certs\") pod \"sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.126741 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.126712 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-home\") pod \"sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.126876 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.126746 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-model-cache\") pod \"sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.126876 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.126838 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnzh2\" (UniqueName: \"kubernetes.io/projected/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-kube-api-access-jnzh2\") pod \"sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.128118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.127678 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-tmp-dir\") pod \"sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.128790 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.128332 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-home\") pod \"sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.128949 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.128924 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-model-cache\") pod \"sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.131454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.131426 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-dshm\") pod \"sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.131973 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.131952 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-tls-certs\") pod \"sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.150412 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.150376 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnzh2\" (UniqueName: \"kubernetes.io/projected/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-kube-api-access-jnzh2\") pod \"sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.300522 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.298975 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:48.585870 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.585783 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:48.707233 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.707206 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r"] Oct 30 02:38:48.709730 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:38:48.709698 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13bc0eed_818a_4ef8_bc7a_6d9a2aed513f.slice/crio-0740ffd990d1007a9915a379f8227cfd2866a32b5b42378db966e3ebcacf75c9 WatchSource:0}: Error finding container 0740ffd990d1007a9915a379f8227cfd2866a32b5b42378db966e3ebcacf75c9: Status 404 returned error can't find the container with id 0740ffd990d1007a9915a379f8227cfd2866a32b5b42378db966e3ebcacf75c9 Oct 30 02:38:48.787538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.787503 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" event={"ID":"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f","Type":"ContainerStarted","Data":"1c6c1bd95fe09c8b35543adea05056e16b5b0c674a9543da57ac934e02a2a234"} Oct 30 02:38:48.787669 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.787556 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" event={"ID":"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f","Type":"ContainerStarted","Data":"0740ffd990d1007a9915a379f8227cfd2866a32b5b42378db966e3ebcacf75c9"} Oct 30 02:38:48.827012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.826961 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" podStartSLOduration=1.826946835 podStartE2EDuration="1.826946835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:38:48.826085885 +0000 UTC m=+4158.404876968" watchObservedRunningTime="2026-10-30 02:38:48.826946835 +0000 UTC m=+4158.405737916" Oct 30 02:38:48.973580 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:48.973057 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:49.300155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:49.300051 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:50.092041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:50.092003 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:51.678841 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:51.678807 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:53.244882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:53.244851 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:54.916680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:54.916644 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:55.227010 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:38:55.226909 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:38:56.481342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:56.481304 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:58.149428 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:58.149381 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:38:58.300156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:58.300040 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:58.313229 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:58.313199 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:58.948061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:58.948015 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:38:59.775820 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:38:59.775778 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:01.035309 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:01.035270 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx"] Oct 30 02:39:01.043976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:01.043941 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ee1fa15-ccbcx"] Oct 30 02:39:01.231998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:01.231772 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:01.362219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:01.362130 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:01.629689 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:01.629585 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:01.826738 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:01.826704 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:02.540062 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:02.540022 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab218198-547c-419f-879f-675b090b1c67" path="/var/lib/kubelet/pods/ab218198-547c-419f-879f-675b090b1c67/volumes" Oct 30 02:39:03.060947 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:03.060911 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:04.757325 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:04.757293 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:06.408495 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:06.408449 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:07.967276 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:07.967229 5998 patch_prober.go:29] interesting pod/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 02:39:07.967690 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:07.967297 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" podUID="dd7460a2-c42d-4d5f-964e-525a49cdafff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 02:39:08.032919 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:08.032873 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:08.843687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:08.843652 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:08.966015 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:08.965938 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:09.016812 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:09.016772 5998 generic.go:356] "Generic (PLEG): container finished" podID="dd7460a2-c42d-4d5f-964e-525a49cdafff" containerID="c0a1c55f47d781e132faae60ef728cf9a1cbff25fc1a3a59920398e2394e9579" exitCode=0 Oct 30 02:39:09.017288 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:09.016844 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" event={"ID":"dd7460a2-c42d-4d5f-964e-525a49cdafff","Type":"ContainerDied","Data":"c0a1c55f47d781e132faae60ef728cf9a1cbff25fc1a3a59920398e2394e9579"} Oct 30 02:39:09.017414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:09.017395 5998 scope.go:122] "RemoveContainer" containerID="c0a1c55f47d781e132faae60ef728cf9a1cbff25fc1a3a59920398e2394e9579" Oct 30 02:39:09.019456 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:09.019421 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:39:09.019576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:09.019511 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:39:09.019576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:09.019561 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:39:09.325563 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:09.325472 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:09.577515 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:09.577425 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:10.052168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:10.052038 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" event={"ID":"dd7460a2-c42d-4d5f-964e-525a49cdafff","Type":"ContainerStarted","Data":"bcbe76c268002a902325d0ad0509f0227b841373a59020e0025d2ad863b82989"} Oct 30 02:39:10.465588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:10.465544 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:39:10.465818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:10.465794 5998 patch_prober.go:29] interesting pod/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Oct 30 02:39:10.465906 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:10.465862 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" podUID="dd7460a2-c42d-4d5f-964e-525a49cdafff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Oct 30 02:39:11.188510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:11.188463 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:11.465886 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:11.465797 5998 patch_prober.go:29] interesting pod/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Oct 30 02:39:11.465886 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:11.465854 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" podUID="dd7460a2-c42d-4d5f-964e-525a49cdafff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Oct 30 02:39:12.047499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:12.047452 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx"] Oct 30 02:39:12.057316 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:12.057279 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7aa71fe-lmtqx"] Oct 30 02:39:12.466339 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:12.466302 5998 patch_prober.go:29] interesting pod/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Oct 30 02:39:12.466870 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:12.466367 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" podUID="dd7460a2-c42d-4d5f-964e-525a49cdafff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Oct 30 02:39:12.547438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:12.547397 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64ea060-92e7-4058-a390-f9192c1c879b" path="/var/lib/kubelet/pods/b64ea060-92e7-4058-a390-f9192c1c879b/volumes" Oct 30 02:39:12.786953 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:12.786867 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:13.465954 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:13.465914 5998 patch_prober.go:29] interesting pod/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Oct 30 02:39:13.466206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:13.465974 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" podUID="dd7460a2-c42d-4d5f-964e-525a49cdafff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Oct 30 02:39:14.341758 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.341717 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:14.466183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.466148 5998 patch_prober.go:29] interesting pod/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Oct 30 02:39:14.466372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.466206 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" podUID="dd7460a2-c42d-4d5f-964e-525a49cdafff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Oct 30 02:39:14.599599 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.599509 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf"] Oct 30 02:39:14.612070 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.612038 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.642188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.639635 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9a1cd0\"/\"kube-root-ca.crt\"" Oct 30 02:39:14.642188 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.640772 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9a1cd0\"/\"openshift-service-ca.crt\"" Oct 30 02:39:14.646207 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.644315 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf"] Oct 30 02:39:14.650448 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.650414 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9a1cd0\"/\"default-dockercfg-sj8kl\"" Oct 30 02:39:14.650618 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.650540 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9a1cd0\"/\"sub-test-model-9a1cd0-kserve-self-signed-certs\"" Oct 30 02:39:14.660210 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.660149 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:14.760076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.760029 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7lkf\" (UniqueName: \"kubernetes.io/projected/5f859aa2-e664-4e9b-b8d7-474991da223d-kube-api-access-k7lkf\") pod \"sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.760337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.760163 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-dshm\") pod \"sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.760337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.760218 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-model-cache\") pod \"sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.760337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.760301 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-home\") pod \"sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.760566 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.760353 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-tmp-dir\") pod \"sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.760566 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.760397 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f859aa2-e664-4e9b-b8d7-474991da223d-tls-certs\") pod \"sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.861785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.861680 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-home\") pod \"sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.861970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.861801 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-tmp-dir\") pod \"sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.861970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.861860 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f859aa2-e664-4e9b-b8d7-474991da223d-tls-certs\") pod \"sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.862118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.861973 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7lkf\" (UniqueName: \"kubernetes.io/projected/5f859aa2-e664-4e9b-b8d7-474991da223d-kube-api-access-k7lkf\") pod \"sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.862118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.862038 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-dshm\") pod \"sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.862235 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.862125 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-model-cache\") pod \"sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.862304 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.862244 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-home\") pod \"sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.862304 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.862266 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-tmp-dir\") pod \"sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.862612 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.862567 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-model-cache\") pod \"sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.865253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.865226 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f859aa2-e664-4e9b-b8d7-474991da223d-tls-certs\") pod \"sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.867596 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.867574 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-dshm\") pod \"sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.885158 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.884531 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7lkf\" (UniqueName: \"kubernetes.io/projected/5f859aa2-e664-4e9b-b8d7-474991da223d-kube-api-access-k7lkf\") pod \"sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:14.972000 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:14.965614 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:15.444613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:15.444527 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf"] Oct 30 02:39:15.448771 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:39:15.448735 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f859aa2_e664_4e9b_b8d7_474991da223d.slice/crio-0c368c63ff914345a4ed1c9251564dc361ca62a64ebb086056d94907f2373532 WatchSource:0}: Error finding container 0c368c63ff914345a4ed1c9251564dc361ca62a64ebb086056d94907f2373532: Status 404 returned error can't find the container with id 0c368c63ff914345a4ed1c9251564dc361ca62a64ebb086056d94907f2373532 Oct 30 02:39:15.465797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:15.465763 5998 patch_prober.go:29] interesting pod/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" start-of-body= Oct 30 02:39:15.465931 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:15.465824 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" podUID="dd7460a2-c42d-4d5f-964e-525a49cdafff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.44:15021/healthz/ready\": dial tcp 10.133.1.44:15021: connect: connection refused" Oct 30 02:39:15.975738 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:15.975670 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:16.132845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:16.132803 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" event={"ID":"5f859aa2-e664-4e9b-b8d7-474991da223d","Type":"ContainerStarted","Data":"83836b039d3c67e4479858d1f1fc5ed2cda2052ab5e91adcfffe076d7aec8da6"} Oct 30 02:39:16.133028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:16.132855 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" event={"ID":"5f859aa2-e664-4e9b-b8d7-474991da223d","Type":"ContainerStarted","Data":"0c368c63ff914345a4ed1c9251564dc361ca62a64ebb086056d94907f2373532"} Oct 30 02:39:16.174562 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:16.174470 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" podStartSLOduration=2.174455359 podStartE2EDuration="2.174455359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:39:16.170639738 +0000 UTC m=+4185.749430819" watchObservedRunningTime="2026-10-30 02:39:16.174455359 +0000 UTC m=+4185.753246440" Oct 30 02:39:16.476369 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:16.476329 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:39:16.476817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:16.476645 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:39:16.477370 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:16.477344 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:39:16.966994 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:16.966959 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:17.107730 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:17.107687 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:17.243432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:17.243340 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:17.635621 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:17.635533 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:18.083146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:18.082639 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:18.475422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:18.475382 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:19.527553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:19.527505 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:39:20.172237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:20.172195 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:21.892757 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:21.892717 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:23.454234 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:23.454190 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:24.658498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:24.658453 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:24.966713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:24.966677 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:24.982635 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:24.982602 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:25.194608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:25.194534 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:25.274787 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:39:25.274672 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:39:25.381043 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:25.381000 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:39:26.963627 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:26.963591 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:27.793007 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:27.792971 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:27.903148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:27.903072 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:28.428218 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:28.428179 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:28.796909 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:28.796823 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx"] Oct 30 02:39:28.797147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:28.797114 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" podUID="1dac6d55-be03-4ce9-b8a7-a6397747e73d" containerName="istio-proxy" containerID="cri-o://2c195c6e29b1fbba708584386692afa2293b1bfb8d88c01105bd0f73e498be9d" gracePeriod=30 Oct 30 02:39:28.825576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:28.825531 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:28.949117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:28.948583 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:30.452865 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:30.452823 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:30.593771 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:30.593728 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:31.259354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:31.259320 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:31.368025 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:31.367935 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:31.885763 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:31.885709 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:32.321172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:32.321084 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:33.825883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:33.825847 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:34.098360 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.098332 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:39:34.172994 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.172958 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-podinfo\") pod \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " Oct 30 02:39:34.173228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.173013 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istiod-ca-cert\") pod \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " Oct 30 02:39:34.173228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.173158 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-workload-certs\") pod \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " Oct 30 02:39:34.173228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.173188 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-workload-socket\") pod \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " Oct 30 02:39:34.173228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.173216 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-envoy\") pod \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " Oct 30 02:39:34.173502 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.173262 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-data\") pod \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " Oct 30 02:39:34.173502 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.173297 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-token\" (UniqueName: \"kubernetes.io/projected/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-token\") pod \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " Oct 30 02:39:34.173502 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.173334 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-credential-socket\") pod \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " Oct 30 02:39:34.173502 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.173449 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dac6d55-be03-4ce9-b8a7-a6397747e73d-kube-api-access-w2lxx\" (UniqueName: \"kubernetes.io/projected/1dac6d55-be03-4ce9-b8a7-a6397747e73d-kube-api-access-w2lxx\") pod \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\" (UID: \"1dac6d55-be03-4ce9-b8a7-a6397747e73d\") " Oct 30 02:39:34.173733 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.173544 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istiod-ca-cert" pod "1dac6d55-be03-4ce9-b8a7-a6397747e73d" (UID: "1dac6d55-be03-4ce9-b8a7-a6397747e73d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:39:34.174623 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.174170 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:39:34.174623 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.174433 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-data" pod "1dac6d55-be03-4ce9-b8a7-a6397747e73d" (UID: "1dac6d55-be03-4ce9-b8a7-a6397747e73d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:39:34.174623 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.174533 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-workload-socket" pod "1dac6d55-be03-4ce9-b8a7-a6397747e73d" (UID: "1dac6d55-be03-4ce9-b8a7-a6397747e73d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:39:34.174623 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.174593 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-credential-socket" pod "1dac6d55-be03-4ce9-b8a7-a6397747e73d" (UID: "1dac6d55-be03-4ce9-b8a7-a6397747e73d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:39:34.175949 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.175926 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-workload-certs" pod "1dac6d55-be03-4ce9-b8a7-a6397747e73d" (UID: "1dac6d55-be03-4ce9-b8a7-a6397747e73d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:39:34.179532 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.179506 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-podinfo" pod "1dac6d55-be03-4ce9-b8a7-a6397747e73d" (UID: "1dac6d55-be03-4ce9-b8a7-a6397747e73d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:39:34.179783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.179759 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-envoy" pod "1dac6d55-be03-4ce9-b8a7-a6397747e73d" (UID: "1dac6d55-be03-4ce9-b8a7-a6397747e73d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:39:34.179891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.179860 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dac6d55-be03-4ce9-b8a7-a6397747e73d-kube-api-access-w2lxx" pod "1dac6d55-be03-4ce9-b8a7-a6397747e73d" (UID: "1dac6d55-be03-4ce9-b8a7-a6397747e73d"). InnerVolumeSpecName "kube-api-access-w2lxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:39:34.183554 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.183531 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-token" pod "1dac6d55-be03-4ce9-b8a7-a6397747e73d" (UID: "1dac6d55-be03-4ce9-b8a7-a6397747e73d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:39:34.275887 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.275846 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2lxx\" (UniqueName: \"kubernetes.io/projected/1dac6d55-be03-4ce9-b8a7-a6397747e73d-kube-api-access-w2lxx\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:39:34.275887 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.275883 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:39:34.275887 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.275895 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:39:34.276185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.275905 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:39:34.276185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.275913 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:39:34.276185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.275921 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:39:34.276185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.275929 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dac6d55-be03-4ce9-b8a7-a6397747e73d-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:39:34.276185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.275936 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dac6d55-be03-4ce9-b8a7-a6397747e73d-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:39:34.447910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.447872 5998 generic.go:356] "Generic (PLEG): container finished" podID="1dac6d55-be03-4ce9-b8a7-a6397747e73d" containerID="2c195c6e29b1fbba708584386692afa2293b1bfb8d88c01105bd0f73e498be9d" exitCode=0 Oct 30 02:39:34.448155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.448005 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" Oct 30 02:39:34.448155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.448004 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" event={"ID":"1dac6d55-be03-4ce9-b8a7-a6397747e73d","Type":"ContainerDied","Data":"2c195c6e29b1fbba708584386692afa2293b1bfb8d88c01105bd0f73e498be9d"} Oct 30 02:39:34.448155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.448084 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx" event={"ID":"1dac6d55-be03-4ce9-b8a7-a6397747e73d","Type":"ContainerDied","Data":"8ca010333abe3facf8128abed95933ea85cf83b7d9fbaf8a809dd12034540c18"} Oct 30 02:39:34.448349 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.448159 5998 scope.go:122] "RemoveContainer" containerID="2c195c6e29b1fbba708584386692afa2293b1bfb8d88c01105bd0f73e498be9d" Oct 30 02:39:34.462502 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.462479 5998 scope.go:122] "RemoveContainer" containerID="3d9086dab4379432ea308ad7b591c989fb1b3b1da907058abd6c69aefd966c58" Oct 30 02:39:34.472956 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.472932 5998 scope.go:122] "RemoveContainer" containerID="2c195c6e29b1fbba708584386692afa2293b1bfb8d88c01105bd0f73e498be9d" Oct 30 02:39:34.473239 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:39:34.473215 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c195c6e29b1fbba708584386692afa2293b1bfb8d88c01105bd0f73e498be9d\": container with ID starting with 2c195c6e29b1fbba708584386692afa2293b1bfb8d88c01105bd0f73e498be9d not found: ID does not exist" containerID="2c195c6e29b1fbba708584386692afa2293b1bfb8d88c01105bd0f73e498be9d" Oct 30 02:39:34.473315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.473247 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c195c6e29b1fbba708584386692afa2293b1bfb8d88c01105bd0f73e498be9d"} err="failed to get container status \"2c195c6e29b1fbba708584386692afa2293b1bfb8d88c01105bd0f73e498be9d\": rpc error: code = NotFound desc = could not find container \"2c195c6e29b1fbba708584386692afa2293b1bfb8d88c01105bd0f73e498be9d\": container with ID starting with 2c195c6e29b1fbba708584386692afa2293b1bfb8d88c01105bd0f73e498be9d not found: ID does not exist" Oct 30 02:39:34.473315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.473264 5998 scope.go:122] "RemoveContainer" containerID="3d9086dab4379432ea308ad7b591c989fb1b3b1da907058abd6c69aefd966c58" Oct 30 02:39:34.473506 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:39:34.473488 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d9086dab4379432ea308ad7b591c989fb1b3b1da907058abd6c69aefd966c58\": container with ID starting with 3d9086dab4379432ea308ad7b591c989fb1b3b1da907058abd6c69aefd966c58 not found: ID does not exist" containerID="3d9086dab4379432ea308ad7b591c989fb1b3b1da907058abd6c69aefd966c58" Oct 30 02:39:34.473551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.473511 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d9086dab4379432ea308ad7b591c989fb1b3b1da907058abd6c69aefd966c58"} err="failed to get container status \"3d9086dab4379432ea308ad7b591c989fb1b3b1da907058abd6c69aefd966c58\": rpc error: code = NotFound desc = could not find container \"3d9086dab4379432ea308ad7b591c989fb1b3b1da907058abd6c69aefd966c58\": container with ID starting with 3d9086dab4379432ea308ad7b591c989fb1b3b1da907058abd6c69aefd966c58 not found: ID does not exist" Oct 30 02:39:34.497745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.497681 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx"] Oct 30 02:39:34.511755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.511502 5998 scope.go:122] "RemoveContainer" containerID="62e9b8507f69e73c7e522f4d6fa9262daa6d2ff4ae6d598219f50c54090f8fb5" Oct 30 02:39:34.525916 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.525883 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-07fb25-openshift-default-7d85d646c7-882rx"] Oct 30 02:39:34.535047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.535024 5998 scope.go:122] "RemoveContainer" containerID="cc8c26f63c0f966f8f71bc241fe275f34e98278148c264a8643f6a570a9bb157" Oct 30 02:39:34.542389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.542358 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dac6d55-be03-4ce9-b8a7-a6397747e73d" path="/var/lib/kubelet/pods/1dac6d55-be03-4ce9-b8a7-a6397747e73d/volumes" Oct 30 02:39:34.548819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:34.548798 5998 scope.go:122] "RemoveContainer" containerID="953e0b47c15e8a44542b8d2abafc674f8aa8204d722fce31498c13b239e8f3ca" Oct 30 02:39:35.365657 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:35.365627 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:36.266069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.266029 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:36.371767 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.371731 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:36.497697 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.497657 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq"] Oct 30 02:39:36.498272 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.498254 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1dac6d55-be03-4ce9-b8a7-a6397747e73d" containerName="istio-proxy" Oct 30 02:39:36.498272 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.498272 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dac6d55-be03-4ce9-b8a7-a6397747e73d" containerName="istio-proxy" Oct 30 02:39:36.498433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.498415 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dac6d55-be03-4ce9-b8a7-a6397747e73d" containerName="istio-proxy" Oct 30 02:39:36.498433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.498429 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dac6d55-be03-4ce9-b8a7-a6397747e73d" containerName="istio-proxy" Oct 30 02:39:36.502297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.502270 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq" Oct 30 02:39:36.529375 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.529288 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:39:36.529375 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.529289 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:39:36.529375 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.529288 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:39:36.529375 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.529305 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:39:36.551905 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.551863 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq"] Oct 30 02:39:36.609564 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.609527 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq6p4\" (UniqueName: \"kubernetes.io/projected/de41b56f-ff6f-4e31-af10-acf732942839-kube-api-access-vq6p4\") pod \"maas-api-revoke-sub-b273dc16-8h7xq\" (UID: \"de41b56f-ff6f-4e31-af10-acf732942839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq" Oct 30 02:39:36.609770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.609579 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de41b56f-ff6f-4e31-af10-acf732942839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b273dc16-8h7xq\" (UID: \"de41b56f-ff6f-4e31-af10-acf732942839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq" Oct 30 02:39:36.609841 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.609802 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de41b56f-ff6f-4e31-af10-acf732942839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b273dc16-8h7xq\" (UID: \"de41b56f-ff6f-4e31-af10-acf732942839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq" Oct 30 02:39:36.711617 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.711579 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de41b56f-ff6f-4e31-af10-acf732942839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b273dc16-8h7xq\" (UID: \"de41b56f-ff6f-4e31-af10-acf732942839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq" Oct 30 02:39:36.711849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.711832 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq6p4\" (UniqueName: \"kubernetes.io/projected/de41b56f-ff6f-4e31-af10-acf732942839-kube-api-access-vq6p4\") pod \"maas-api-revoke-sub-b273dc16-8h7xq\" (UID: \"de41b56f-ff6f-4e31-af10-acf732942839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq" Oct 30 02:39:36.711915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.711880 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de41b56f-ff6f-4e31-af10-acf732942839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b273dc16-8h7xq\" (UID: \"de41b56f-ff6f-4e31-af10-acf732942839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq" Oct 30 02:39:36.712874 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.712847 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de41b56f-ff6f-4e31-af10-acf732942839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b273dc16-8h7xq\" (UID: \"de41b56f-ff6f-4e31-af10-acf732942839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq" Oct 30 02:39:36.714489 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.714461 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de41b56f-ff6f-4e31-af10-acf732942839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b273dc16-8h7xq\" (UID: \"de41b56f-ff6f-4e31-af10-acf732942839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq" Oct 30 02:39:36.734462 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.734432 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq6p4\" (UniqueName: \"kubernetes.io/projected/de41b56f-ff6f-4e31-af10-acf732942839-kube-api-access-vq6p4\") pod \"maas-api-revoke-sub-b273dc16-8h7xq\" (UID: \"de41b56f-ff6f-4e31-af10-acf732942839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq" Oct 30 02:39:36.820414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:36.820326 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq" Oct 30 02:39:37.018034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:37.017995 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:37.147577 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:37.147543 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq"] Oct 30 02:39:37.149692 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:39:37.149653 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde41b56f_ff6f_4e31_af10_acf732942839.slice/crio-9b20cb167b1100548df221ed6187370598c0d1e52d2764f0d7efdd050624d72b WatchSource:0}: Error finding container 9b20cb167b1100548df221ed6187370598c0d1e52d2764f0d7efdd050624d72b: Status 404 returned error can't find the container with id 9b20cb167b1100548df221ed6187370598c0d1e52d2764f0d7efdd050624d72b Oct 30 02:39:37.498429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:37.498393 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq" event={"ID":"de41b56f-ff6f-4e31-af10-acf732942839","Type":"ContainerStarted","Data":"625e032a79654547409ed683c8678a1c9db2080f89fc507c57a73dd5e3a1fd84"} Oct 30 02:39:37.498904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:37.498446 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq" event={"ID":"de41b56f-ff6f-4e31-af10-acf732942839","Type":"ContainerStarted","Data":"9b20cb167b1100548df221ed6187370598c0d1e52d2764f0d7efdd050624d72b"} Oct 30 02:39:37.538595 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:37.538525 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq" podStartSLOduration=1.538505814 podStartE2EDuration="1.538505814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:39:37.536376176 +0000 UTC m=+4207.115167257" watchObservedRunningTime="2026-10-30 02:39:37.538505814 +0000 UTC m=+4207.117296896" Oct 30 02:39:38.380318 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:38.380220 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:38.513285 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:38.513241 5998 generic.go:356] "Generic (PLEG): container finished" podID="de41b56f-ff6f-4e31-af10-acf732942839" containerID="625e032a79654547409ed683c8678a1c9db2080f89fc507c57a73dd5e3a1fd84" exitCode=0 Oct 30 02:39:38.513843 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:38.513291 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq" event={"ID":"de41b56f-ff6f-4e31-af10-acf732942839","Type":"ContainerDied","Data":"625e032a79654547409ed683c8678a1c9db2080f89fc507c57a73dd5e3a1fd84"} Oct 30 02:39:38.889114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:38.883595 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:39.134467 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:39.134428 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:39.675890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:39.675859 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq" Oct 30 02:39:39.800355 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:39.800262 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de41b56f-ff6f-4e31-af10-acf732942839-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de41b56f-ff6f-4e31-af10-acf732942839-maas-api-openshift-ca\") pod \"de41b56f-ff6f-4e31-af10-acf732942839\" (UID: \"de41b56f-ff6f-4e31-af10-acf732942839\") " Oct 30 02:39:39.800355 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:39.800308 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de41b56f-ff6f-4e31-af10-acf732942839-kube-api-access-vq6p4\" (UniqueName: \"kubernetes.io/projected/de41b56f-ff6f-4e31-af10-acf732942839-kube-api-access-vq6p4\") pod \"de41b56f-ff6f-4e31-af10-acf732942839\" (UID: \"de41b56f-ff6f-4e31-af10-acf732942839\") " Oct 30 02:39:39.800355 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:39.800338 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de41b56f-ff6f-4e31-af10-acf732942839-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de41b56f-ff6f-4e31-af10-acf732942839-maas-api-xks-ca\") pod \"de41b56f-ff6f-4e31-af10-acf732942839\" (UID: \"de41b56f-ff6f-4e31-af10-acf732942839\") " Oct 30 02:39:39.800721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:39.800689 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de41b56f-ff6f-4e31-af10-acf732942839-maas-api-openshift-ca" pod "de41b56f-ff6f-4e31-af10-acf732942839" (UID: "de41b56f-ff6f-4e31-af10-acf732942839"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:39:39.800984 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:39.800965 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de41b56f-ff6f-4e31-af10-acf732942839-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:39:39.802532 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:39.802501 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de41b56f-ff6f-4e31-af10-acf732942839-maas-api-xks-ca" pod "de41b56f-ff6f-4e31-af10-acf732942839" (UID: "de41b56f-ff6f-4e31-af10-acf732942839"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:39:39.802716 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:39.802696 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de41b56f-ff6f-4e31-af10-acf732942839-kube-api-access-vq6p4" pod "de41b56f-ff6f-4e31-af10-acf732942839" (UID: "de41b56f-ff6f-4e31-af10-acf732942839"). InnerVolumeSpecName "kube-api-access-vq6p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:39:39.902106 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:39.902059 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq6p4\" (UniqueName: \"kubernetes.io/projected/de41b56f-ff6f-4e31-af10-acf732942839-kube-api-access-vq6p4\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:39:39.902106 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:39.902106 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de41b56f-ff6f-4e31-af10-acf732942839-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:39:40.530076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:40.530040 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq" Oct 30 02:39:40.530076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:40.530058 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq" event={"ID":"de41b56f-ff6f-4e31-af10-acf732942839","Type":"ContainerDied","Data":"9b20cb167b1100548df221ed6187370598c0d1e52d2764f0d7efdd050624d72b"} Oct 30 02:39:40.530383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:40.530128 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b20cb167b1100548df221ed6187370598c0d1e52d2764f0d7efdd050624d72b" Oct 30 02:39:40.703173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:40.703043 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:40.818693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:40.818583 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:42.224124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:42.224055 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:42.367509 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:42.367468 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:42.501083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:42.500992 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:42.907141 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:42.907015 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:43.272484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:43.272446 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:44.137506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:44.137458 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:44.434170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:44.434060 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:46.156058 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:46.156021 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:46.886393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:46.886357 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:47.694581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:47.694544 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:49.354445 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:49.354393 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:49.477805 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:49.477759 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:50.963362 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:50.963317 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:51.048655 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:51.048502 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6"] Oct 30 02:39:51.054381 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:51.054329 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4"] Oct 30 02:39:51.060320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:51.060284 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e30d778-s6wt6"] Oct 30 02:39:51.063882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:51.063856 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0443d77-6zkh4"] Oct 30 02:39:51.324865 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:51.324770 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:52.541073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:52.541037 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="650423de-a897-4e26-b2a8-c822c8642895" path="/var/lib/kubelet/pods/650423de-a897-4e26-b2a8-c822c8642895/volumes" Oct 30 02:39:52.541615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:52.541598 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a76fafcf-a716-4a3b-86fb-a12a17f1ae5e" path="/var/lib/kubelet/pods/a76fafcf-a716-4a3b-86fb-a12a17f1ae5e/volumes" Oct 30 02:39:52.573193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:52.573151 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:53.467243 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:53.467203 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:53.944286 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:53.944231 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:54.683719 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:54.683683 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:55.219351 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:39:55.219304 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:39:56.174633 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:56.174593 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:57.690335 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:57.690294 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:58.553693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:58.553656 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:58.662780 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:58.662741 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:59.280957 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:59.280917 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:59.395321 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:59.395288 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:39:59.642745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:59.642642 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h"] Oct 30 02:39:59.644513 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:59.644471 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" podUID="8532131f-abe1-426d-9a42-36ad42fb7cdb" containerName="istio-proxy" containerID="cri-o://2c760d1ac0802012f6dc21673b956e3af438973144bdc4d7def29f34f1257338" gracePeriod=30 Oct 30 02:39:59.672627 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:39:59.672587 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:00.418191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:00.418152 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:01.063566 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:01.063525 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j"] Oct 30 02:40:01.073194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:01.073160 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-987cba5c-zl58j"] Oct 30 02:40:01.097363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:01.097320 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:02.037160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:02.037121 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5"] Oct 30 02:40:02.045339 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:02.045301 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65479906-qqbt5"] Oct 30 02:40:02.540617 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:02.540573 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fd3f37a-562d-4b69-b061-7c26891d0c43" path="/var/lib/kubelet/pods/1fd3f37a-562d-4b69-b061-7c26891d0c43/volumes" Oct 30 02:40:02.541209 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:02.541187 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a10a5bfc-b78e-4075-b194-71c14f8e0a80" path="/var/lib/kubelet/pods/a10a5bfc-b78e-4075-b194-71c14f8e0a80/volumes" Oct 30 02:40:02.710192 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:02.710149 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:04.298181 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:04.298142 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:04.942937 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:04.942905 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:40:05.108796 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.108743 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-podinfo\") pod \"8532131f-abe1-426d-9a42-36ad42fb7cdb\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " Oct 30 02:40:05.109133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.108865 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-envoy\") pod \"8532131f-abe1-426d-9a42-36ad42fb7cdb\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " Oct 30 02:40:05.109133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.108943 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8532131f-abe1-426d-9a42-36ad42fb7cdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8532131f-abe1-426d-9a42-36ad42fb7cdb-istiod-ca-cert\") pod \"8532131f-abe1-426d-9a42-36ad42fb7cdb\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " Oct 30 02:40:05.109133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.108987 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-credential-socket\") pod \"8532131f-abe1-426d-9a42-36ad42fb7cdb\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " Oct 30 02:40:05.110707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.109360 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-credential-socket" pod "8532131f-abe1-426d-9a42-36ad42fb7cdb" (UID: "8532131f-abe1-426d-9a42-36ad42fb7cdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:05.110707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.109458 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-data\") pod \"8532131f-abe1-426d-9a42-36ad42fb7cdb\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " Oct 30 02:40:05.110707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.109568 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-token\" (UniqueName: \"kubernetes.io/projected/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-token\") pod \"8532131f-abe1-426d-9a42-36ad42fb7cdb\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " Oct 30 02:40:05.110707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.109647 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8532131f-abe1-426d-9a42-36ad42fb7cdb-istiod-ca-cert" pod "8532131f-abe1-426d-9a42-36ad42fb7cdb" (UID: "8532131f-abe1-426d-9a42-36ad42fb7cdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:40:05.110707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.109661 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-workload-certs\") pod \"8532131f-abe1-426d-9a42-36ad42fb7cdb\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " Oct 30 02:40:05.110707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.109705 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8532131f-abe1-426d-9a42-36ad42fb7cdb-kube-api-access-sbpbz\" (UniqueName: \"kubernetes.io/projected/8532131f-abe1-426d-9a42-36ad42fb7cdb-kube-api-access-sbpbz\") pod \"8532131f-abe1-426d-9a42-36ad42fb7cdb\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " Oct 30 02:40:05.110707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.109735 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-workload-socket\") pod \"8532131f-abe1-426d-9a42-36ad42fb7cdb\" (UID: \"8532131f-abe1-426d-9a42-36ad42fb7cdb\") " Oct 30 02:40:05.110707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.109835 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-data" pod "8532131f-abe1-426d-9a42-36ad42fb7cdb" (UID: "8532131f-abe1-426d-9a42-36ad42fb7cdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:05.110707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.110084 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-workload-certs" pod "8532131f-abe1-426d-9a42-36ad42fb7cdb" (UID: "8532131f-abe1-426d-9a42-36ad42fb7cdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:05.110707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.110345 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-workload-socket" pod "8532131f-abe1-426d-9a42-36ad42fb7cdb" (UID: "8532131f-abe1-426d-9a42-36ad42fb7cdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:05.110707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.110614 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:40:05.110707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.110636 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:40:05.110707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.110652 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:40:05.110707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.110667 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:40:05.110707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.110680 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8532131f-abe1-426d-9a42-36ad42fb7cdb-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:40:05.111981 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.111522 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-envoy" pod "8532131f-abe1-426d-9a42-36ad42fb7cdb" (UID: "8532131f-abe1-426d-9a42-36ad42fb7cdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:40:05.111981 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.111637 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-podinfo" pod "8532131f-abe1-426d-9a42-36ad42fb7cdb" (UID: "8532131f-abe1-426d-9a42-36ad42fb7cdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:40:05.112257 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.112220 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-token" pod "8532131f-abe1-426d-9a42-36ad42fb7cdb" (UID: "8532131f-abe1-426d-9a42-36ad42fb7cdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:40:05.112547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.112526 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8532131f-abe1-426d-9a42-36ad42fb7cdb-kube-api-access-sbpbz" pod "8532131f-abe1-426d-9a42-36ad42fb7cdb" (UID: "8532131f-abe1-426d-9a42-36ad42fb7cdb"). InnerVolumeSpecName "kube-api-access-sbpbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:40:05.212699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.212591 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbpbz\" (UniqueName: \"kubernetes.io/projected/8532131f-abe1-426d-9a42-36ad42fb7cdb-kube-api-access-sbpbz\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:40:05.212699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.212637 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:40:05.212699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.212651 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:40:05.212699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.212663 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8532131f-abe1-426d-9a42-36ad42fb7cdb-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:40:05.766187 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.766148 5998 generic.go:356] "Generic (PLEG): container finished" podID="8532131f-abe1-426d-9a42-36ad42fb7cdb" containerID="2c760d1ac0802012f6dc21673b956e3af438973144bdc4d7def29f34f1257338" exitCode=0 Oct 30 02:40:05.766624 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.766196 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" Oct 30 02:40:05.766624 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.766217 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" event={"ID":"8532131f-abe1-426d-9a42-36ad42fb7cdb","Type":"ContainerDied","Data":"2c760d1ac0802012f6dc21673b956e3af438973144bdc4d7def29f34f1257338"} Oct 30 02:40:05.766624 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.766273 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h" event={"ID":"8532131f-abe1-426d-9a42-36ad42fb7cdb","Type":"ContainerDied","Data":"82a991226617a7db43864bb8680e640f877b7b14311a32ce313518ecc6cec3db"} Oct 30 02:40:05.766624 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.766296 5998 scope.go:122] "RemoveContainer" containerID="2c760d1ac0802012f6dc21673b956e3af438973144bdc4d7def29f34f1257338" Oct 30 02:40:05.778282 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.778260 5998 scope.go:122] "RemoveContainer" containerID="3cd7d398c5dc61ee87456fa1d7adbb155d01ac3f9c9d421d2f637ac3c163a3c5" Oct 30 02:40:05.790147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.790117 5998 scope.go:122] "RemoveContainer" containerID="2c760d1ac0802012f6dc21673b956e3af438973144bdc4d7def29f34f1257338" Oct 30 02:40:05.790456 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:40:05.790431 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c760d1ac0802012f6dc21673b956e3af438973144bdc4d7def29f34f1257338\": container with ID starting with 2c760d1ac0802012f6dc21673b956e3af438973144bdc4d7def29f34f1257338 not found: ID does not exist" containerID="2c760d1ac0802012f6dc21673b956e3af438973144bdc4d7def29f34f1257338" Oct 30 02:40:05.790518 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.790466 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c760d1ac0802012f6dc21673b956e3af438973144bdc4d7def29f34f1257338"} err="failed to get container status \"2c760d1ac0802012f6dc21673b956e3af438973144bdc4d7def29f34f1257338\": rpc error: code = NotFound desc = could not find container \"2c760d1ac0802012f6dc21673b956e3af438973144bdc4d7def29f34f1257338\": container with ID starting with 2c760d1ac0802012f6dc21673b956e3af438973144bdc4d7def29f34f1257338 not found: ID does not exist" Oct 30 02:40:05.790518 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.790483 5998 scope.go:122] "RemoveContainer" containerID="3cd7d398c5dc61ee87456fa1d7adbb155d01ac3f9c9d421d2f637ac3c163a3c5" Oct 30 02:40:05.790796 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:40:05.790772 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cd7d398c5dc61ee87456fa1d7adbb155d01ac3f9c9d421d2f637ac3c163a3c5\": container with ID starting with 3cd7d398c5dc61ee87456fa1d7adbb155d01ac3f9c9d421d2f637ac3c163a3c5 not found: ID does not exist" containerID="3cd7d398c5dc61ee87456fa1d7adbb155d01ac3f9c9d421d2f637ac3c163a3c5" Oct 30 02:40:05.790840 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.790804 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cd7d398c5dc61ee87456fa1d7adbb155d01ac3f9c9d421d2f637ac3c163a3c5"} err="failed to get container status \"3cd7d398c5dc61ee87456fa1d7adbb155d01ac3f9c9d421d2f637ac3c163a3c5\": rpc error: code = NotFound desc = could not find container \"3cd7d398c5dc61ee87456fa1d7adbb155d01ac3f9c9d421d2f637ac3c163a3c5\": container with ID starting with 3cd7d398c5dc61ee87456fa1d7adbb155d01ac3f9c9d421d2f637ac3c163a3c5 not found: ID does not exist" Oct 30 02:40:05.826579 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.826535 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h"] Oct 30 02:40:05.839258 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.839223 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-93ec46-openshift-default-8d6db8b57-nhh7h"] Oct 30 02:40:05.875474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:05.875438 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:06.564770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:06.564732 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:06.567673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:06.567639 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8532131f-abe1-426d-9a42-36ad42fb7cdb" path="/var/lib/kubelet/pods/8532131f-abe1-426d-9a42-36ad42fb7cdb/volumes" Oct 30 02:40:06.677684 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:06.677643 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:07.599377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:07.599341 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:07.834714 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:07.834675 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:08.678821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:08.678782 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:08.785951 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:08.785914 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:09.276698 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:09.276660 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:10.869758 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:10.869721 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:12.398252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:12.398166 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:14.009281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:14.009237 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:14.865121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:14.865070 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:40:15.669033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:15.668994 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:15.814793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:15.814756 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:17.179257 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:17.179206 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:40:17.214307 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:17.214240 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:40:17.216157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:17.216121 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:40:17.371112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:17.370218 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" podUID="5f859aa2-e664-4e9b-b8d7-474991da223d" containerName="main" probeResult="failure" output="Get \"https://10.133.1.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:40:17.485696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:17.485599 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:17.507428 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:17.507381 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:40:17.916315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:17.916158 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" podUID="13bc0eed-818a-4ef8-bc7a-6d9a2aed513f" containerName="main" probeResult="failure" output="Get \"https://10.133.1.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:40:19.270378 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:19.270342 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:20.999822 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:20.999777 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:22.652827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:22.652784 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:24.174981 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:24.174948 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:25.218510 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:40:25.218467 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:40:25.488500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:25.488404 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:26.200081 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:26.200046 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:27.897548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:27.897512 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:29.437415 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:29.437373 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:31.003506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:31.003463 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:32.580371 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:32.580326 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:32.719112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:32.714237 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:33.260780 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:33.260747 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:33.696978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:33.696939 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:34.347898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:34.347837 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:34.456881 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:34.456843 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:34.596078 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:34.596042 5998 scope.go:122] "RemoveContainer" containerID="19421242744bb7d3a935abd1bb73e0021e1c13534250a5a2baf8de347a6fbfcf" Oct 30 02:40:34.607971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:34.607946 5998 scope.go:122] "RemoveContainer" containerID="0c35026dceaa208d7b499a98bd5c0bf5fe40644416287b485d4b2c86224ff2c4" Oct 30 02:40:36.106552 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:36.106516 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:37.736651 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:37.736608 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:39.630563 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:39.630523 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:40.377449 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:40.377416 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:40.507450 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:40.506456 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:40.890780 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:40.890742 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:41.546186 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:41.546137 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:42.851897 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:42.851858 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:42.979394 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:42.979357 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:43.130630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:43.130533 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:44.711332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:44.711295 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:46.314100 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:46.314052 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:48.004928 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:48.004883 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:48.246693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:48.246649 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:48.437933 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:48.437892 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:49.792665 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:49.792621 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:51.215199 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:51.215160 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:51.525590 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:51.525497 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:51.784925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:51.784835 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:53.271612 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:53.271580 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:54.923025 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:54.922987 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:55.229028 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:40:55.228925 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:40:56.500996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:56.500963 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:58.079149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:58.079114 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:58.253381 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:58.253343 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:40:59.474477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:40:59.474434 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:00.135264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:00.135228 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:01.867139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:01.867073 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:02.693228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:02.693193 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:03.589429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:03.589390 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:05.168253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:05.168214 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:05.289402 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:05.289365 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:05.405740 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:05.405704 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:05.885029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:05.884993 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:06.360369 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:06.360333 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw"] Oct 30 02:41:06.361237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:06.361168 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" podUID="d71cb5c7-54c2-42ed-9d33-de325c6c851a" containerName="istio-proxy" containerID="cri-o://11757bda7031d9f39d2f6e41e45495bee41e05f4bf0628b31bd9754ac1f6cbfb" gracePeriod=30 Oct 30 02:41:06.395666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:06.395631 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:07.116860 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:07.116811 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:07.235875 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:07.235315 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:07.358826 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:07.358782 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:41:07.359037 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:07.358887 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" podUID="d71cb5c7-54c2-42ed-9d33-de325c6c851a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:41:08.469576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:08.469539 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:08.677039 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:08.676997 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:08.870345 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:08.870248 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:10.513784 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:10.513750 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:10.623236 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:10.623190 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:11.674197 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.674169 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:41:11.711580 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.711544 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-data\") pod \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " Oct 30 02:41:11.711787 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.711612 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-workload-socket\") pod \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " Oct 30 02:41:11.711787 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.711644 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-token\" (UniqueName: \"kubernetes.io/projected/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-token\") pod \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " Oct 30 02:41:11.711787 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.711727 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-workload-certs\") pod \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " Oct 30 02:41:11.711956 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.711863 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-podinfo\") pod \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " Oct 30 02:41:11.711956 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.711902 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-data" pod "d71cb5c7-54c2-42ed-9d33-de325c6c851a" (UID: "d71cb5c7-54c2-42ed-9d33-de325c6c851a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:11.712080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.711969 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-workload-socket" pod "d71cb5c7-54c2-42ed-9d33-de325c6c851a" (UID: "d71cb5c7-54c2-42ed-9d33-de325c6c851a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:11.712080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.712012 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-workload-certs" pod "d71cb5c7-54c2-42ed-9d33-de325c6c851a" (UID: "d71cb5c7-54c2-42ed-9d33-de325c6c851a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:11.712080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.712042 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istiod-ca-cert\") pod \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " Oct 30 02:41:11.712288 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.712112 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-envoy\") pod \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " Oct 30 02:41:11.712362 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.712345 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istiod-ca-cert" pod "d71cb5c7-54c2-42ed-9d33-de325c6c851a" (UID: "d71cb5c7-54c2-42ed-9d33-de325c6c851a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:11.712473 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.712450 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-credential-socket" pod "d71cb5c7-54c2-42ed-9d33-de325c6c851a" (UID: "d71cb5c7-54c2-42ed-9d33-de325c6c851a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:11.712545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.712493 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-credential-socket\") pod \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " Oct 30 02:41:11.712545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.712528 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71cb5c7-54c2-42ed-9d33-de325c6c851a-kube-api-access-5kljp\" (UniqueName: \"kubernetes.io/projected/d71cb5c7-54c2-42ed-9d33-de325c6c851a-kube-api-access-5kljp\") pod \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\" (UID: \"d71cb5c7-54c2-42ed-9d33-de325c6c851a\") " Oct 30 02:41:11.713287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.713263 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:11.713433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.713287 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:11.713433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.713302 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:11.713433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.713314 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:11.713433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.713335 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:11.714454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.714427 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-token" pod "d71cb5c7-54c2-42ed-9d33-de325c6c851a" (UID: "d71cb5c7-54c2-42ed-9d33-de325c6c851a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:11.714627 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.714603 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-envoy" pod "d71cb5c7-54c2-42ed-9d33-de325c6c851a" (UID: "d71cb5c7-54c2-42ed-9d33-de325c6c851a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:11.714705 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.714682 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-podinfo" pod "d71cb5c7-54c2-42ed-9d33-de325c6c851a" (UID: "d71cb5c7-54c2-42ed-9d33-de325c6c851a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:41:11.714842 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.714817 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71cb5c7-54c2-42ed-9d33-de325c6c851a-kube-api-access-5kljp" pod "d71cb5c7-54c2-42ed-9d33-de325c6c851a" (UID: "d71cb5c7-54c2-42ed-9d33-de325c6c851a"). InnerVolumeSpecName "kube-api-access-5kljp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:11.814474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.814438 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:11.814474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.814470 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:11.814707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.814491 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d71cb5c7-54c2-42ed-9d33-de325c6c851a-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:11.814707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.814503 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kljp\" (UniqueName: \"kubernetes.io/projected/d71cb5c7-54c2-42ed-9d33-de325c6c851a-kube-api-access-5kljp\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:11.973518 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:11.973485 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:12.542881 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:12.542841 5998 generic.go:356] "Generic (PLEG): container finished" podID="d71cb5c7-54c2-42ed-9d33-de325c6c851a" containerID="11757bda7031d9f39d2f6e41e45495bee41e05f4bf0628b31bd9754ac1f6cbfb" exitCode=0 Oct 30 02:41:12.543217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:12.542964 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" Oct 30 02:41:12.546221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:12.546190 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" event={"ID":"d71cb5c7-54c2-42ed-9d33-de325c6c851a","Type":"ContainerDied","Data":"11757bda7031d9f39d2f6e41e45495bee41e05f4bf0628b31bd9754ac1f6cbfb"} Oct 30 02:41:12.546363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:12.546250 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw" event={"ID":"d71cb5c7-54c2-42ed-9d33-de325c6c851a","Type":"ContainerDied","Data":"fea45f3f4fef9f785ddcb3f343b0c1fba793367923cc2521671eaedf1d2c9d44"} Oct 30 02:41:12.546363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:12.546273 5998 scope.go:122] "RemoveContainer" containerID="11757bda7031d9f39d2f6e41e45495bee41e05f4bf0628b31bd9754ac1f6cbfb" Oct 30 02:41:12.562320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:12.562292 5998 scope.go:122] "RemoveContainer" containerID="ec6c449f8666fee5516ac536d6fc4ee05fe5053a80fe1061479126cf82b278b1" Oct 30 02:41:12.572439 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:12.572419 5998 scope.go:122] "RemoveContainer" containerID="11757bda7031d9f39d2f6e41e45495bee41e05f4bf0628b31bd9754ac1f6cbfb" Oct 30 02:41:12.572729 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:41:12.572708 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11757bda7031d9f39d2f6e41e45495bee41e05f4bf0628b31bd9754ac1f6cbfb\": container with ID starting with 11757bda7031d9f39d2f6e41e45495bee41e05f4bf0628b31bd9754ac1f6cbfb not found: ID does not exist" containerID="11757bda7031d9f39d2f6e41e45495bee41e05f4bf0628b31bd9754ac1f6cbfb" Oct 30 02:41:12.572785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:12.572738 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11757bda7031d9f39d2f6e41e45495bee41e05f4bf0628b31bd9754ac1f6cbfb"} err="failed to get container status \"11757bda7031d9f39d2f6e41e45495bee41e05f4bf0628b31bd9754ac1f6cbfb\": rpc error: code = NotFound desc = could not find container \"11757bda7031d9f39d2f6e41e45495bee41e05f4bf0628b31bd9754ac1f6cbfb\": container with ID starting with 11757bda7031d9f39d2f6e41e45495bee41e05f4bf0628b31bd9754ac1f6cbfb not found: ID does not exist" Oct 30 02:41:12.572785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:12.572754 5998 scope.go:122] "RemoveContainer" containerID="ec6c449f8666fee5516ac536d6fc4ee05fe5053a80fe1061479126cf82b278b1" Oct 30 02:41:12.573015 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:41:12.572993 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6c449f8666fee5516ac536d6fc4ee05fe5053a80fe1061479126cf82b278b1\": container with ID starting with ec6c449f8666fee5516ac536d6fc4ee05fe5053a80fe1061479126cf82b278b1 not found: ID does not exist" containerID="ec6c449f8666fee5516ac536d6fc4ee05fe5053a80fe1061479126cf82b278b1" Oct 30 02:41:12.573067 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:12.573020 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6c449f8666fee5516ac536d6fc4ee05fe5053a80fe1061479126cf82b278b1"} err="failed to get container status \"ec6c449f8666fee5516ac536d6fc4ee05fe5053a80fe1061479126cf82b278b1\": rpc error: code = NotFound desc = could not find container \"ec6c449f8666fee5516ac536d6fc4ee05fe5053a80fe1061479126cf82b278b1\": container with ID starting with ec6c449f8666fee5516ac536d6fc4ee05fe5053a80fe1061479126cf82b278b1 not found: ID does not exist" Oct 30 02:41:12.616162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:12.616122 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw"] Oct 30 02:41:12.627939 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:12.627904 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-0ce950-openshift-default-88864987c-5sxfw"] Oct 30 02:41:12.629325 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:12.629295 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:13.280139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:13.280076 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:13.476298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:13.476261 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:13.836664 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:13.836623 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:41:14.540123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:14.540065 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d71cb5c7-54c2-42ed-9d33-de325c6c851a" path="/var/lib/kubelet/pods/d71cb5c7-54c2-42ed-9d33-de325c6c851a/volumes" Oct 30 02:41:14.801713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:14.801624 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:15.038485 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:15.038438 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t"] Oct 30 02:41:15.065168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:15.064639 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfcc483a-6ss2t"] Oct 30 02:41:16.359513 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:16.359475 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:16.489247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:16.489210 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:16.553225 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:16.553189 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e352468-3606-467c-abd3-4654142859b0" path="/var/lib/kubelet/pods/7e352468-3606-467c-abd3-4654142859b0/volumes" Oct 30 02:41:18.295328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:18.295291 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:18.764125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:18.764075 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:19.379339 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:19.379304 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:20.197938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:20.197900 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:21.131707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:21.131668 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:21.253625 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:21.253561 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:21.367322 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:21.367285 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:21.870638 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:21.870597 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:22.150033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:22.149943 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:22.491516 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:22.491471 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:22.928747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:22.928707 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:23.327454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.327368 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc"] Oct 30 02:41:23.328180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.328156 5998 cpu_manager.go:414] "containerMap: removing container" podUID="d71cb5c7-54c2-42ed-9d33-de325c6c851a" containerName="istio-proxy" Oct 30 02:41:23.328301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.328181 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71cb5c7-54c2-42ed-9d33-de325c6c851a" containerName="istio-proxy" Oct 30 02:41:23.328301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.328197 5998 cpu_manager.go:414] "containerMap: removing container" podUID="de41b56f-ff6f-4e31-af10-acf732942839" containerName="revoke-keys" Oct 30 02:41:23.328301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.328205 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de41b56f-ff6f-4e31-af10-acf732942839" containerName="revoke-keys" Oct 30 02:41:23.328301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.328221 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8532131f-abe1-426d-9a42-36ad42fb7cdb" containerName="istio-proxy" Oct 30 02:41:23.328301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.328228 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8532131f-abe1-426d-9a42-36ad42fb7cdb" containerName="istio-proxy" Oct 30 02:41:23.328301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.328254 5998 cpu_manager.go:414] "containerMap: removing container" podUID="d71cb5c7-54c2-42ed-9d33-de325c6c851a" containerName="istio-proxy" Oct 30 02:41:23.328301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.328263 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71cb5c7-54c2-42ed-9d33-de325c6c851a" containerName="istio-proxy" Oct 30 02:41:23.328301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.328278 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1dac6d55-be03-4ce9-b8a7-a6397747e73d" containerName="istio-proxy" Oct 30 02:41:23.328301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.328285 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dac6d55-be03-4ce9-b8a7-a6397747e73d" containerName="istio-proxy" Oct 30 02:41:23.328796 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.328479 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71cb5c7-54c2-42ed-9d33-de325c6c851a" containerName="istio-proxy" Oct 30 02:41:23.328796 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.328493 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="de41b56f-ff6f-4e31-af10-acf732942839" containerName="revoke-keys" Oct 30 02:41:23.328796 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.328504 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8532131f-abe1-426d-9a42-36ad42fb7cdb" containerName="istio-proxy" Oct 30 02:41:23.328796 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.328516 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71cb5c7-54c2-42ed-9d33-de325c6c851a" containerName="istio-proxy" Oct 30 02:41:23.328796 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.328527 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8532131f-abe1-426d-9a42-36ad42fb7cdb" containerName="istio-proxy" Oct 30 02:41:23.333324 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.333300 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc" Oct 30 02:41:23.346229 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.346192 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:41:23.356516 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.356481 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:41:23.356673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.356542 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:41:23.356673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.356602 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:41:23.361028 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.360998 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc"] Oct 30 02:41:23.404358 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.404322 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00355226-9a13-4568-8152-76c07ac13698-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-869e9f05-mxpwc\" (UID: \"00355226-9a13-4568-8152-76c07ac13698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc" Oct 30 02:41:23.404549 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.404477 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00355226-9a13-4568-8152-76c07ac13698-maas-api-xks-ca\") pod \"maas-api-revoke-sub-869e9f05-mxpwc\" (UID: \"00355226-9a13-4568-8152-76c07ac13698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc" Oct 30 02:41:23.404640 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.404605 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9pjw\" (UniqueName: \"kubernetes.io/projected/00355226-9a13-4568-8152-76c07ac13698-kube-api-access-v9pjw\") pod \"maas-api-revoke-sub-869e9f05-mxpwc\" (UID: \"00355226-9a13-4568-8152-76c07ac13698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc" Oct 30 02:41:23.505866 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.505819 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00355226-9a13-4568-8152-76c07ac13698-maas-api-xks-ca\") pod \"maas-api-revoke-sub-869e9f05-mxpwc\" (UID: \"00355226-9a13-4568-8152-76c07ac13698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc" Oct 30 02:41:23.506122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.505973 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9pjw\" (UniqueName: \"kubernetes.io/projected/00355226-9a13-4568-8152-76c07ac13698-kube-api-access-v9pjw\") pod \"maas-api-revoke-sub-869e9f05-mxpwc\" (UID: \"00355226-9a13-4568-8152-76c07ac13698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc" Oct 30 02:41:23.506122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.506102 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00355226-9a13-4568-8152-76c07ac13698-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-869e9f05-mxpwc\" (UID: \"00355226-9a13-4568-8152-76c07ac13698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc" Oct 30 02:41:23.506920 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.506892 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00355226-9a13-4568-8152-76c07ac13698-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-869e9f05-mxpwc\" (UID: \"00355226-9a13-4568-8152-76c07ac13698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc" Oct 30 02:41:23.508437 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.508416 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00355226-9a13-4568-8152-76c07ac13698-maas-api-xks-ca\") pod \"maas-api-revoke-sub-869e9f05-mxpwc\" (UID: \"00355226-9a13-4568-8152-76c07ac13698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc" Oct 30 02:41:23.527933 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.527903 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9pjw\" (UniqueName: \"kubernetes.io/projected/00355226-9a13-4568-8152-76c07ac13698-kube-api-access-v9pjw\") pod \"maas-api-revoke-sub-869e9f05-mxpwc\" (UID: \"00355226-9a13-4568-8152-76c07ac13698\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc" Oct 30 02:41:23.647907 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.647816 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc" Oct 30 02:41:23.780714 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.780676 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:23.972259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:23.972226 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc"] Oct 30 02:41:23.974143 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:41:23.974064 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00355226_9a13_4568_8152_76c07ac13698.slice/crio-46d64caf63a5730c4497b0fa10a93c68bcaa792774fa7d4d9318a6059b8805e8 WatchSource:0}: Error finding container 46d64caf63a5730c4497b0fa10a93c68bcaa792774fa7d4d9318a6059b8805e8: Status 404 returned error can't find the container with id 46d64caf63a5730c4497b0fa10a93c68bcaa792774fa7d4d9318a6059b8805e8 Oct 30 02:41:24.675266 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:24.675228 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc" event={"ID":"00355226-9a13-4568-8152-76c07ac13698","Type":"ContainerStarted","Data":"72b6955641d324c3848a8fbd1ca29f7c9eecae83bcdb37b7a69e987328391d20"} Oct 30 02:41:24.675789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:24.675293 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc" event={"ID":"00355226-9a13-4568-8152-76c07ac13698","Type":"ContainerStarted","Data":"46d64caf63a5730c4497b0fa10a93c68bcaa792774fa7d4d9318a6059b8805e8"} Oct 30 02:41:24.718576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:24.718516 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc" podStartSLOduration=1.718495753 podStartE2EDuration="1.718495753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:24.714351956 +0000 UTC m=+4314.293143038" watchObservedRunningTime="2026-10-30 02:41:24.718495753 +0000 UTC m=+4314.297286839" Oct 30 02:41:24.779431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:24.779395 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:25.180663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:25.180620 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:25.234662 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:41:25.234565 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:41:25.745431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:25.745389 5998 generic.go:356] "Generic (PLEG): container finished" podID="00355226-9a13-4568-8152-76c07ac13698" containerID="72b6955641d324c3848a8fbd1ca29f7c9eecae83bcdb37b7a69e987328391d20" exitCode=0 Oct 30 02:41:25.745997 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:25.745555 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc" event={"ID":"00355226-9a13-4568-8152-76c07ac13698","Type":"ContainerDied","Data":"72b6955641d324c3848a8fbd1ca29f7c9eecae83bcdb37b7a69e987328391d20"} Oct 30 02:41:26.009930 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:26.009836 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:26.934253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:26.934220 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc" Oct 30 02:41:26.974889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:26.974851 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00355226-9a13-4568-8152-76c07ac13698-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00355226-9a13-4568-8152-76c07ac13698-maas-api-openshift-ca\") pod \"00355226-9a13-4568-8152-76c07ac13698\" (UID: \"00355226-9a13-4568-8152-76c07ac13698\") " Oct 30 02:41:26.975144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:26.974916 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00355226-9a13-4568-8152-76c07ac13698-kube-api-access-v9pjw\" (UniqueName: \"kubernetes.io/projected/00355226-9a13-4568-8152-76c07ac13698-kube-api-access-v9pjw\") pod \"00355226-9a13-4568-8152-76c07ac13698\" (UID: \"00355226-9a13-4568-8152-76c07ac13698\") " Oct 30 02:41:26.975144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:26.975016 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00355226-9a13-4568-8152-76c07ac13698-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00355226-9a13-4568-8152-76c07ac13698-maas-api-xks-ca\") pod \"00355226-9a13-4568-8152-76c07ac13698\" (UID: \"00355226-9a13-4568-8152-76c07ac13698\") " Oct 30 02:41:26.975318 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:26.975290 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00355226-9a13-4568-8152-76c07ac13698-maas-api-openshift-ca" pod "00355226-9a13-4568-8152-76c07ac13698" (UID: "00355226-9a13-4568-8152-76c07ac13698"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:26.975685 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:26.975642 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00355226-9a13-4568-8152-76c07ac13698-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:26.977890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:26.977778 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00355226-9a13-4568-8152-76c07ac13698-kube-api-access-v9pjw" pod "00355226-9a13-4568-8152-76c07ac13698" (UID: "00355226-9a13-4568-8152-76c07ac13698"). InnerVolumeSpecName "kube-api-access-v9pjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:26.977890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:26.977863 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00355226-9a13-4568-8152-76c07ac13698-maas-api-xks-ca" pod "00355226-9a13-4568-8152-76c07ac13698" (UID: "00355226-9a13-4568-8152-76c07ac13698"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:27.077130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:27.077038 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00355226-9a13-4568-8152-76c07ac13698-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:27.077130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:27.077072 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9pjw\" (UniqueName: \"kubernetes.io/projected/00355226-9a13-4568-8152-76c07ac13698-kube-api-access-v9pjw\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:27.486762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:27.486728 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:27.686098 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:27.686050 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:27.767131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:27.767024 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc" event={"ID":"00355226-9a13-4568-8152-76c07ac13698","Type":"ContainerDied","Data":"46d64caf63a5730c4497b0fa10a93c68bcaa792774fa7d4d9318a6059b8805e8"} Oct 30 02:41:27.767131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:27.767056 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc" Oct 30 02:41:27.767131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:27.767078 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46d64caf63a5730c4497b0fa10a93c68bcaa792774fa7d4d9318a6059b8805e8" Oct 30 02:41:27.804039 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:27.804006 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:27.951759 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:27.951722 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv"] Oct 30 02:41:28.053384 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:28.053297 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:28.365671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:28.365589 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:28.746320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:28.746277 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx"] Oct 30 02:41:28.746623 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:28.746569 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" podUID="dd7460a2-c42d-4d5f-964e-525a49cdafff" containerName="istio-proxy" containerID="cri-o://bcbe76c268002a902325d0ad0509f0227b841373a59020e0025d2ad863b82989" gracePeriod=30 Oct 30 02:41:28.777907 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:28.777877 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:29.282661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:29.282623 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:31.081258 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:31.081222 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:31.478990 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:31.478947 5998 patch_prober.go:29] interesting pod/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:41:31.479219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:31.479024 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" podUID="dd7460a2-c42d-4d5f-964e-525a49cdafff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:41:32.707741 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:32.707705 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:34.029854 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.029789 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:41:34.077001 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.076964 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd7460a2-c42d-4d5f-964e-525a49cdafff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7460a2-c42d-4d5f-964e-525a49cdafff-istiod-ca-cert\") pod \"dd7460a2-c42d-4d5f-964e-525a49cdafff\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " Oct 30 02:41:34.077228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.077018 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-envoy\") pod \"dd7460a2-c42d-4d5f-964e-525a49cdafff\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " Oct 30 02:41:34.077228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.077057 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-data\") pod \"dd7460a2-c42d-4d5f-964e-525a49cdafff\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " Oct 30 02:41:34.077228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.077126 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-token\" (UniqueName: \"kubernetes.io/projected/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-token\") pod \"dd7460a2-c42d-4d5f-964e-525a49cdafff\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " Oct 30 02:41:34.077419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.077258 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-workload-certs\") pod \"dd7460a2-c42d-4d5f-964e-525a49cdafff\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " Oct 30 02:41:34.077419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.077357 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-podinfo\") pod \"dd7460a2-c42d-4d5f-964e-525a49cdafff\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " Oct 30 02:41:34.077419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.077415 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-credential-socket\") pod \"dd7460a2-c42d-4d5f-964e-525a49cdafff\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " Oct 30 02:41:34.077605 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.077453 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-workload-socket\") pod \"dd7460a2-c42d-4d5f-964e-525a49cdafff\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " Oct 30 02:41:34.077605 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.077509 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd7460a2-c42d-4d5f-964e-525a49cdafff-kube-api-access-rfs27\" (UniqueName: \"kubernetes.io/projected/dd7460a2-c42d-4d5f-964e-525a49cdafff-kube-api-access-rfs27\") pod \"dd7460a2-c42d-4d5f-964e-525a49cdafff\" (UID: \"dd7460a2-c42d-4d5f-964e-525a49cdafff\") " Oct 30 02:41:34.077752 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.077446 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-data" pod "dd7460a2-c42d-4d5f-964e-525a49cdafff" (UID: "dd7460a2-c42d-4d5f-964e-525a49cdafff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:34.077752 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.077619 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd7460a2-c42d-4d5f-964e-525a49cdafff-istiod-ca-cert" pod "dd7460a2-c42d-4d5f-964e-525a49cdafff" (UID: "dd7460a2-c42d-4d5f-964e-525a49cdafff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:34.077752 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.077622 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-workload-certs" pod "dd7460a2-c42d-4d5f-964e-525a49cdafff" (UID: "dd7460a2-c42d-4d5f-964e-525a49cdafff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:34.077752 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.077669 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-credential-socket" pod "dd7460a2-c42d-4d5f-964e-525a49cdafff" (UID: "dd7460a2-c42d-4d5f-964e-525a49cdafff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:34.077987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.077788 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-workload-socket" pod "dd7460a2-c42d-4d5f-964e-525a49cdafff" (UID: "dd7460a2-c42d-4d5f-964e-525a49cdafff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:34.078139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.078078 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:34.078272 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.078143 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:34.078272 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.078158 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd7460a2-c42d-4d5f-964e-525a49cdafff-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:34.078272 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.078169 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:34.078272 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.078180 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:34.079654 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.079623 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-token" pod "dd7460a2-c42d-4d5f-964e-525a49cdafff" (UID: "dd7460a2-c42d-4d5f-964e-525a49cdafff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:34.079862 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.079834 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-podinfo" pod "dd7460a2-c42d-4d5f-964e-525a49cdafff" (UID: "dd7460a2-c42d-4d5f-964e-525a49cdafff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:41:34.079948 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.079877 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-envoy" pod "dd7460a2-c42d-4d5f-964e-525a49cdafff" (UID: "dd7460a2-c42d-4d5f-964e-525a49cdafff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:34.080053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.080034 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7460a2-c42d-4d5f-964e-525a49cdafff-kube-api-access-rfs27" pod "dd7460a2-c42d-4d5f-964e-525a49cdafff" (UID: "dd7460a2-c42d-4d5f-964e-525a49cdafff"). InnerVolumeSpecName "kube-api-access-rfs27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:34.179273 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.179238 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:34.179273 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.179274 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfs27\" (UniqueName: \"kubernetes.io/projected/dd7460a2-c42d-4d5f-964e-525a49cdafff-kube-api-access-rfs27\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:34.179499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.179291 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:34.179499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.179306 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd7460a2-c42d-4d5f-964e-525a49cdafff-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:34.362663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.362571 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:34.656563 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.656467 5998 scope.go:122] "RemoveContainer" containerID="89b2eb57d9d808784e44714c8cf40cd787bbdf56cfe0e9bb7ba6be8a6db62cde" Oct 30 02:41:34.667947 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.667916 5998 scope.go:122] "RemoveContainer" containerID="5f9dc054e7db368be5c33f46b7b2ee4502b5b3eabe309d2103213fcf16e99415" Oct 30 02:41:34.679647 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.679617 5998 scope.go:122] "RemoveContainer" containerID="58a6254e95a0af3ab30624c257380d9079bd470d2e6f90ae506ec047b1f3dd30" Oct 30 02:41:34.691665 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.691640 5998 scope.go:122] "RemoveContainer" containerID="5036786471da0ccef4efbd4f2ad228c4b137d01fdec12980f404e9d4ea036cff" Oct 30 02:41:34.823575 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.823539 5998 generic.go:356] "Generic (PLEG): container finished" podID="dd7460a2-c42d-4d5f-964e-525a49cdafff" containerID="bcbe76c268002a902325d0ad0509f0227b841373a59020e0025d2ad863b82989" exitCode=0 Oct 30 02:41:34.823800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.823615 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" Oct 30 02:41:34.823800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.823624 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" event={"ID":"dd7460a2-c42d-4d5f-964e-525a49cdafff","Type":"ContainerDied","Data":"bcbe76c268002a902325d0ad0509f0227b841373a59020e0025d2ad863b82989"} Oct 30 02:41:34.823800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.823685 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx" event={"ID":"dd7460a2-c42d-4d5f-964e-525a49cdafff","Type":"ContainerDied","Data":"d88d545e00226a0380765c959edf5b08069888eaf469146c695a882f352348f9"} Oct 30 02:41:34.823800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.823730 5998 scope.go:122] "RemoveContainer" containerID="bcbe76c268002a902325d0ad0509f0227b841373a59020e0025d2ad863b82989" Oct 30 02:41:34.834492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.834471 5998 scope.go:122] "RemoveContainer" containerID="c0a1c55f47d781e132faae60ef728cf9a1cbff25fc1a3a59920398e2394e9579" Oct 30 02:41:34.844246 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.844217 5998 scope.go:122] "RemoveContainer" containerID="bcbe76c268002a902325d0ad0509f0227b841373a59020e0025d2ad863b82989" Oct 30 02:41:34.844518 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:41:34.844494 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcbe76c268002a902325d0ad0509f0227b841373a59020e0025d2ad863b82989\": container with ID starting with bcbe76c268002a902325d0ad0509f0227b841373a59020e0025d2ad863b82989 not found: ID does not exist" containerID="bcbe76c268002a902325d0ad0509f0227b841373a59020e0025d2ad863b82989" Oct 30 02:41:34.844567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.844531 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcbe76c268002a902325d0ad0509f0227b841373a59020e0025d2ad863b82989"} err="failed to get container status \"bcbe76c268002a902325d0ad0509f0227b841373a59020e0025d2ad863b82989\": rpc error: code = NotFound desc = could not find container \"bcbe76c268002a902325d0ad0509f0227b841373a59020e0025d2ad863b82989\": container with ID starting with bcbe76c268002a902325d0ad0509f0227b841373a59020e0025d2ad863b82989 not found: ID does not exist" Oct 30 02:41:34.844567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.844555 5998 scope.go:122] "RemoveContainer" containerID="c0a1c55f47d781e132faae60ef728cf9a1cbff25fc1a3a59920398e2394e9579" Oct 30 02:41:34.844803 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:41:34.844782 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a1c55f47d781e132faae60ef728cf9a1cbff25fc1a3a59920398e2394e9579\": container with ID starting with c0a1c55f47d781e132faae60ef728cf9a1cbff25fc1a3a59920398e2394e9579 not found: ID does not exist" containerID="c0a1c55f47d781e132faae60ef728cf9a1cbff25fc1a3a59920398e2394e9579" Oct 30 02:41:34.844869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.844813 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a1c55f47d781e132faae60ef728cf9a1cbff25fc1a3a59920398e2394e9579"} err="failed to get container status \"c0a1c55f47d781e132faae60ef728cf9a1cbff25fc1a3a59920398e2394e9579\": rpc error: code = NotFound desc = could not find container \"c0a1c55f47d781e132faae60ef728cf9a1cbff25fc1a3a59920398e2394e9579\": container with ID starting with c0a1c55f47d781e132faae60ef728cf9a1cbff25fc1a3a59920398e2394e9579 not found: ID does not exist" Oct 30 02:41:34.882320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.882288 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx"] Oct 30 02:41:34.891953 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:34.891924 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-edae4317-openshift-default-598d57ccbf-5brqx"] Oct 30 02:41:35.862161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:35.862127 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:36.540010 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:36.539975 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd7460a2-c42d-4d5f-964e-525a49cdafff" path="/var/lib/kubelet/pods/dd7460a2-c42d-4d5f-964e-525a49cdafff/volumes" Oct 30 02:41:37.485936 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:37.485904 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:37.856603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:37.856512 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:39.185724 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:39.185687 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:40.795962 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:40.795921 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:40.986462 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:40.986421 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:41.676553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:41.676474 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:42.343025 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:42.342984 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:42.974112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:42.974047 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" containerID="cri-o://e465c9888d7256a975fce10ef6bbdca209528c2d3a1fbeb5104578c317116b19" gracePeriod=45 Oct 30 02:41:43.269461 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.269432 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 02:41:43.422488 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.422442 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4786840-065b-49eb-854c-87ecd6b08916-kube-api-access-vhcx4\" (UniqueName: \"kubernetes.io/projected/e4786840-065b-49eb-854c-87ecd6b08916-kube-api-access-vhcx4\") pod \"e4786840-065b-49eb-854c-87ecd6b08916\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " Oct 30 02:41:43.423206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.423181 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-model-cache\") pod \"e4786840-065b-49eb-854c-87ecd6b08916\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " Oct 30 02:41:43.423685 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.423550 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-model-cache" pod "e4786840-065b-49eb-854c-87ecd6b08916" (UID: "e4786840-065b-49eb-854c-87ecd6b08916"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:43.423874 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.423858 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-home\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-home\") pod \"e4786840-065b-49eb-854c-87ecd6b08916\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " Oct 30 02:41:43.424360 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.424236 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-home" pod "e4786840-065b-49eb-854c-87ecd6b08916" (UID: "e4786840-065b-49eb-854c-87ecd6b08916"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:43.424685 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.424658 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-tmp-dir" pod "e4786840-065b-49eb-854c-87ecd6b08916" (UID: "e4786840-065b-49eb-854c-87ecd6b08916"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:43.424794 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.424779 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-tmp-dir\") pod \"e4786840-065b-49eb-854c-87ecd6b08916\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " Oct 30 02:41:43.424974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.424961 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-dshm\") pod \"e4786840-065b-49eb-854c-87ecd6b08916\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " Oct 30 02:41:43.425515 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.425487 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4786840-065b-49eb-854c-87ecd6b08916-kube-api-access-vhcx4" pod "e4786840-065b-49eb-854c-87ecd6b08916" (UID: "e4786840-065b-49eb-854c-87ecd6b08916"). InnerVolumeSpecName "kube-api-access-vhcx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:43.425674 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.425654 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4786840-065b-49eb-854c-87ecd6b08916-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4786840-065b-49eb-854c-87ecd6b08916-tls-certs\") pod \"e4786840-065b-49eb-854c-87ecd6b08916\" (UID: \"e4786840-065b-49eb-854c-87ecd6b08916\") " Oct 30 02:41:43.426424 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.426404 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhcx4\" (UniqueName: \"kubernetes.io/projected/e4786840-065b-49eb-854c-87ecd6b08916-kube-api-access-vhcx4\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.426550 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.426537 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.426639 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.426629 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.426739 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.426725 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.429299 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.429268 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-dshm" pod "e4786840-065b-49eb-854c-87ecd6b08916" (UID: "e4786840-065b-49eb-854c-87ecd6b08916"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:43.445220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.445188 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4786840-065b-49eb-854c-87ecd6b08916-tls-certs" pod "e4786840-065b-49eb-854c-87ecd6b08916" (UID: "e4786840-065b-49eb-854c-87ecd6b08916"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:43.528528 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.528439 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4786840-065b-49eb-854c-87ecd6b08916-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.528528 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.528467 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4786840-065b-49eb-854c-87ecd6b08916-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:43.781166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.781056 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:43.818679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.818634 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh"] Oct 30 02:41:43.819332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.819311 5998 cpu_manager.go:414] "containerMap: removing container" podUID="dd7460a2-c42d-4d5f-964e-525a49cdafff" containerName="istio-proxy" Oct 30 02:41:43.819332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.819331 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7460a2-c42d-4d5f-964e-525a49cdafff" containerName="istio-proxy" Oct 30 02:41:43.819512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.819357 5998 cpu_manager.go:414] "containerMap: removing container" podUID="00355226-9a13-4568-8152-76c07ac13698" containerName="revoke-keys" Oct 30 02:41:43.819512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.819363 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00355226-9a13-4568-8152-76c07ac13698" containerName="revoke-keys" Oct 30 02:41:43.819512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.819389 5998 cpu_manager.go:414] "containerMap: removing container" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" Oct 30 02:41:43.819512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.819394 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" Oct 30 02:41:43.819512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.819402 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8532131f-abe1-426d-9a42-36ad42fb7cdb" containerName="istio-proxy" Oct 30 02:41:43.819512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.819408 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8532131f-abe1-426d-9a42-36ad42fb7cdb" containerName="istio-proxy" Oct 30 02:41:43.819512 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.819509 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="00355226-9a13-4568-8152-76c07ac13698" containerName="revoke-keys" Oct 30 02:41:43.819872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.819520 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7460a2-c42d-4d5f-964e-525a49cdafff" containerName="istio-proxy" Oct 30 02:41:43.819872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.819534 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4786840-065b-49eb-854c-87ecd6b08916" containerName="main" Oct 30 02:41:43.823206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.823182 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh" Oct 30 02:41:43.834268 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.834211 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mplp\" (UniqueName: \"kubernetes.io/projected/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-kube-api-access-6mplp\") pod \"maas-api-revoke-sub-9b7a1b68-w7glh\" (UID: \"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh" Oct 30 02:41:43.834431 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.834408 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b7a1b68-w7glh\" (UID: \"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh" Oct 30 02:41:43.834505 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.834467 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b7a1b68-w7glh\" (UID: \"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh" Oct 30 02:41:43.836994 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.836969 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:41:43.837264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.837239 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:41:43.847779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.847165 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:41:43.847779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.847180 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:41:43.851466 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.851442 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh"] Oct 30 02:41:43.932336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.932297 5998 generic.go:356] "Generic (PLEG): container finished" podID="e4786840-065b-49eb-854c-87ecd6b08916" containerID="e465c9888d7256a975fce10ef6bbdca209528c2d3a1fbeb5104578c317116b19" exitCode=0 Oct 30 02:41:43.932538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.932341 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" Oct 30 02:41:43.932538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.932378 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" event={"ID":"e4786840-065b-49eb-854c-87ecd6b08916","Type":"ContainerDied","Data":"e465c9888d7256a975fce10ef6bbdca209528c2d3a1fbeb5104578c317116b19"} Oct 30 02:41:43.932538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.932429 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv" event={"ID":"e4786840-065b-49eb-854c-87ecd6b08916","Type":"ContainerDied","Data":"dec74e26eca41dd1368a729ca3526f1470bbf69883125c82eb763d839f0b1f9e"} Oct 30 02:41:43.932538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.932453 5998 scope.go:122] "RemoveContainer" containerID="e465c9888d7256a975fce10ef6bbdca209528c2d3a1fbeb5104578c317116b19" Oct 30 02:41:43.939882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.939832 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b7a1b68-w7glh\" (UID: \"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh" Oct 30 02:41:43.940045 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.939917 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b7a1b68-w7glh\" (UID: \"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh" Oct 30 02:41:43.940045 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.940029 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mplp\" (UniqueName: \"kubernetes.io/projected/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-kube-api-access-6mplp\") pod \"maas-api-revoke-sub-9b7a1b68-w7glh\" (UID: \"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh" Oct 30 02:41:43.941401 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.941197 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b7a1b68-w7glh\" (UID: \"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh" Oct 30 02:41:43.944537 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.944512 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b7a1b68-w7glh\" (UID: \"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh" Oct 30 02:41:43.945801 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.945773 5998 scope.go:122] "RemoveContainer" containerID="e465c9888d7256a975fce10ef6bbdca209528c2d3a1fbeb5104578c317116b19" Oct 30 02:41:43.946164 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:41:43.946136 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e465c9888d7256a975fce10ef6bbdca209528c2d3a1fbeb5104578c317116b19\": container with ID starting with e465c9888d7256a975fce10ef6bbdca209528c2d3a1fbeb5104578c317116b19 not found: ID does not exist" containerID="e465c9888d7256a975fce10ef6bbdca209528c2d3a1fbeb5104578c317116b19" Oct 30 02:41:43.946249 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.946176 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e465c9888d7256a975fce10ef6bbdca209528c2d3a1fbeb5104578c317116b19"} err="failed to get container status \"e465c9888d7256a975fce10ef6bbdca209528c2d3a1fbeb5104578c317116b19\": rpc error: code = NotFound desc = could not find container \"e465c9888d7256a975fce10ef6bbdca209528c2d3a1fbeb5104578c317116b19\": container with ID starting with e465c9888d7256a975fce10ef6bbdca209528c2d3a1fbeb5104578c317116b19 not found: ID does not exist" Oct 30 02:41:43.963945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:43.963907 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mplp\" (UniqueName: \"kubernetes.io/projected/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-kube-api-access-6mplp\") pod \"maas-api-revoke-sub-9b7a1b68-w7glh\" (UID: \"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh" Oct 30 02:41:44.039170 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:44.031152 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv"] Oct 30 02:41:44.055932 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:44.055894 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b93d51/simulator-w4-b93d51-kserve-66ff9f64f9-cwhpv"] Oct 30 02:41:44.138804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:44.138765 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh" Oct 30 02:41:44.457113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:44.453216 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:44.519111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:44.518449 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh"] Oct 30 02:41:44.552401 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:44.552312 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4786840-065b-49eb-854c-87ecd6b08916" path="/var/lib/kubelet/pods/e4786840-065b-49eb-854c-87ecd6b08916/volumes" Oct 30 02:41:44.943152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:44.943103 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh" event={"ID":"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7","Type":"ContainerStarted","Data":"c2e7ca79a0e3ebc7ebbb2bd85f615d30b45ecdc9458a738b68acac6f95e8b39f"} Oct 30 02:41:44.943298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:44.943170 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh" event={"ID":"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7","Type":"ContainerStarted","Data":"9968f977d8a25ed89169b350c3663d027c51176470974118b00316b1f0146723"} Oct 30 02:41:44.981776 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:44.981721 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh" podStartSLOduration=1.981706462 podStartE2EDuration="1.981706462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:41:44.980353137 +0000 UTC m=+4334.559144217" watchObservedRunningTime="2026-10-30 02:41:44.981706462 +0000 UTC m=+4334.560497543" Oct 30 02:41:45.634942 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:45.634777 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:45.756150 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:45.754966 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:45.955060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:45.955024 5998 generic.go:356] "Generic (PLEG): container finished" podID="8b5bea28-5420-4913-a4bb-eb8bbf18dbb7" containerID="c2e7ca79a0e3ebc7ebbb2bd85f615d30b45ecdc9458a738b68acac6f95e8b39f" exitCode=0 Oct 30 02:41:45.955256 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:45.955122 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh" event={"ID":"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7","Type":"ContainerDied","Data":"c2e7ca79a0e3ebc7ebbb2bd85f615d30b45ecdc9458a738b68acac6f95e8b39f"} Oct 30 02:41:46.292185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:46.292044 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:47.133428 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:47.133400 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh" Oct 30 02:41:47.281327 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:47.281235 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-maas-api-openshift-ca\") pod \"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7\" (UID: \"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7\") " Oct 30 02:41:47.281506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:47.281411 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-kube-api-access-6mplp\" (UniqueName: \"kubernetes.io/projected/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-kube-api-access-6mplp\") pod \"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7\" (UID: \"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7\") " Oct 30 02:41:47.281600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:47.281539 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-maas-api-xks-ca\") pod \"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7\" (UID: \"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7\") " Oct 30 02:41:47.281699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:47.281674 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-maas-api-openshift-ca" pod "8b5bea28-5420-4913-a4bb-eb8bbf18dbb7" (UID: "8b5bea28-5420-4913-a4bb-eb8bbf18dbb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:47.281934 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:47.281915 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.283615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:47.283591 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-kube-api-access-6mplp" pod "8b5bea28-5420-4913-a4bb-eb8bbf18dbb7" (UID: "8b5bea28-5420-4913-a4bb-eb8bbf18dbb7"). InnerVolumeSpecName "kube-api-access-6mplp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:47.283680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:47.283634 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-maas-api-xks-ca" pod "8b5bea28-5420-4913-a4bb-eb8bbf18dbb7" (UID: "8b5bea28-5420-4913-a4bb-eb8bbf18dbb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:41:47.383265 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:47.383229 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.383265 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:47.383263 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mplp\" (UniqueName: \"kubernetes.io/projected/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7-kube-api-access-6mplp\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:47.941427 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:47.941385 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:41:47.973035 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:47.973000 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh" Oct 30 02:41:47.973270 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:47.973033 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh" event={"ID":"8b5bea28-5420-4913-a4bb-eb8bbf18dbb7","Type":"ContainerDied","Data":"9968f977d8a25ed89169b350c3663d027c51176470974118b00316b1f0146723"} Oct 30 02:41:47.973270 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:47.973115 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9968f977d8a25ed89169b350c3663d027c51176470974118b00316b1f0146723" Oct 30 02:41:48.155311 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:48.155276 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:49.764979 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:49.764938 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:51.348184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:51.348141 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:52.355104 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:52.355047 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:52.490234 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:52.490203 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:53.224984 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:53.224947 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:53.350204 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:53.350157 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:53.604772 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:53.604684 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb"] Oct 30 02:41:53.605209 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:53.604944 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" podUID="5a29d403-91c3-4ab3-9f67-6d94fcfb557f" containerName="istio-proxy" containerID="cri-o://13b9b2e42b55b77b27259ea8d1a40b49d67703430e513b6907007b9afadf9d9f" gracePeriod=30 Oct 30 02:41:53.643484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:53.643446 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:54.134497 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:54.134462 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:54.347084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:54.347022 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:54.469289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:54.469240 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:55.087034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:55.086980 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:55.225136 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:41:55.225073 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:41:56.708909 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:56.708857 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:58.356494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.356455 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:41:58.605833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.605791 5998 patch_prober.go:29] interesting pod/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:41:58.606006 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.605849 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" podUID="5a29d403-91c3-4ab3-9f67-6d94fcfb557f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:41:58.882618 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.882555 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:41:58.970018 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.969981 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istiod-ca-cert\") pod \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " Oct 30 02:41:58.970264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.970034 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-token\" (UniqueName: \"kubernetes.io/projected/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-token\") pod \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " Oct 30 02:41:58.970264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.970059 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-podinfo\") pod \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " Oct 30 02:41:58.970264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.970159 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-credential-socket\") pod \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " Oct 30 02:41:58.970264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.970186 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-kube-api-access-2l88f\" (UniqueName: \"kubernetes.io/projected/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-kube-api-access-2l88f\") pod \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " Oct 30 02:41:58.970264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.970232 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-envoy\") pod \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " Oct 30 02:41:58.970575 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.970273 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-workload-socket\") pod \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " Oct 30 02:41:58.970652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.970470 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istiod-ca-cert" pod "5a29d403-91c3-4ab3-9f67-6d94fcfb557f" (UID: "5a29d403-91c3-4ab3-9f67-6d94fcfb557f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:41:58.970652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.970573 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-credential-socket" pod "5a29d403-91c3-4ab3-9f67-6d94fcfb557f" (UID: "5a29d403-91c3-4ab3-9f67-6d94fcfb557f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:58.970652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.970441 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-data\") pod \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " Oct 30 02:41:58.970813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.970674 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-data" pod "5a29d403-91c3-4ab3-9f67-6d94fcfb557f" (UID: "5a29d403-91c3-4ab3-9f67-6d94fcfb557f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:58.970956 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.970802 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-workload-socket" pod "5a29d403-91c3-4ab3-9f67-6d94fcfb557f" (UID: "5a29d403-91c3-4ab3-9f67-6d94fcfb557f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:58.971351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.971320 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:58.971464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.971354 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:58.971464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.971369 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:58.971464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.971384 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:58.972529 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.972496 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-podinfo" pod "5a29d403-91c3-4ab3-9f67-6d94fcfb557f" (UID: "5a29d403-91c3-4ab3-9f67-6d94fcfb557f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:41:58.972529 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.972504 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-envoy" pod "5a29d403-91c3-4ab3-9f67-6d94fcfb557f" (UID: "5a29d403-91c3-4ab3-9f67-6d94fcfb557f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:58.972678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.972608 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-token" pod "5a29d403-91c3-4ab3-9f67-6d94fcfb557f" (UID: "5a29d403-91c3-4ab3-9f67-6d94fcfb557f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:58.972678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:58.972647 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-kube-api-access-2l88f" pod "5a29d403-91c3-4ab3-9f67-6d94fcfb557f" (UID: "5a29d403-91c3-4ab3-9f67-6d94fcfb557f"). InnerVolumeSpecName "kube-api-access-2l88f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:41:59.069647 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.069605 5998 generic.go:356] "Generic (PLEG): container finished" podID="5a29d403-91c3-4ab3-9f67-6d94fcfb557f" containerID="13b9b2e42b55b77b27259ea8d1a40b49d67703430e513b6907007b9afadf9d9f" exitCode=0 Oct 30 02:41:59.069843 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.069678 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" Oct 30 02:41:59.069843 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.069698 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" event={"ID":"5a29d403-91c3-4ab3-9f67-6d94fcfb557f","Type":"ContainerDied","Data":"13b9b2e42b55b77b27259ea8d1a40b49d67703430e513b6907007b9afadf9d9f"} Oct 30 02:41:59.069843 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.069774 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb" event={"ID":"5a29d403-91c3-4ab3-9f67-6d94fcfb557f","Type":"ContainerDied","Data":"96c1404051f6058aa87b6a8d9d8c88346177822240f98a490804a45a22bb1397"} Oct 30 02:41:59.069843 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.069807 5998 scope.go:122] "RemoveContainer" containerID="13b9b2e42b55b77b27259ea8d1a40b49d67703430e513b6907007b9afadf9d9f" Oct 30 02:41:59.072795 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.072763 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-workload-certs\") pod \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\" (UID: \"5a29d403-91c3-4ab3-9f67-6d94fcfb557f\") " Oct 30 02:41:59.073069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.073042 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-workload-certs" pod "5a29d403-91c3-4ab3-9f67-6d94fcfb557f" (UID: "5a29d403-91c3-4ab3-9f67-6d94fcfb557f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:41:59.073401 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.073379 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l88f\" (UniqueName: \"kubernetes.io/projected/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-kube-api-access-2l88f\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:59.073476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.073405 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:59.073476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.073419 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:59.073476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.073431 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:59.073476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.073443 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a29d403-91c3-4ab3-9f67-6d94fcfb557f-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:41:59.084322 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.084222 5998 scope.go:122] "RemoveContainer" containerID="30897745dbb277e9949903b625671acd23a5b6555c79a8faecc52b2c6bc8bf30" Oct 30 02:41:59.097149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.097126 5998 scope.go:122] "RemoveContainer" containerID="13b9b2e42b55b77b27259ea8d1a40b49d67703430e513b6907007b9afadf9d9f" Oct 30 02:41:59.097435 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:41:59.097404 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b9b2e42b55b77b27259ea8d1a40b49d67703430e513b6907007b9afadf9d9f\": container with ID starting with 13b9b2e42b55b77b27259ea8d1a40b49d67703430e513b6907007b9afadf9d9f not found: ID does not exist" containerID="13b9b2e42b55b77b27259ea8d1a40b49d67703430e513b6907007b9afadf9d9f" Oct 30 02:41:59.097539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.097441 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b9b2e42b55b77b27259ea8d1a40b49d67703430e513b6907007b9afadf9d9f"} err="failed to get container status \"13b9b2e42b55b77b27259ea8d1a40b49d67703430e513b6907007b9afadf9d9f\": rpc error: code = NotFound desc = could not find container \"13b9b2e42b55b77b27259ea8d1a40b49d67703430e513b6907007b9afadf9d9f\": container with ID starting with 13b9b2e42b55b77b27259ea8d1a40b49d67703430e513b6907007b9afadf9d9f not found: ID does not exist" Oct 30 02:41:59.097539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.097462 5998 scope.go:122] "RemoveContainer" containerID="30897745dbb277e9949903b625671acd23a5b6555c79a8faecc52b2c6bc8bf30" Oct 30 02:41:59.097715 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:41:59.097696 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30897745dbb277e9949903b625671acd23a5b6555c79a8faecc52b2c6bc8bf30\": container with ID starting with 30897745dbb277e9949903b625671acd23a5b6555c79a8faecc52b2c6bc8bf30 not found: ID does not exist" containerID="30897745dbb277e9949903b625671acd23a5b6555c79a8faecc52b2c6bc8bf30" Oct 30 02:41:59.097767 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.097722 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30897745dbb277e9949903b625671acd23a5b6555c79a8faecc52b2c6bc8bf30"} err="failed to get container status \"30897745dbb277e9949903b625671acd23a5b6555c79a8faecc52b2c6bc8bf30\": rpc error: code = NotFound desc = could not find container \"30897745dbb277e9949903b625671acd23a5b6555c79a8faecc52b2c6bc8bf30\": container with ID starting with 30897745dbb277e9949903b625671acd23a5b6555c79a8faecc52b2c6bc8bf30 not found: ID does not exist" Oct 30 02:41:59.414467 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.414428 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb"] Oct 30 02:41:59.429037 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:41:59.428992 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-000edd-openshift-default-6dfb97cbcc-mvdfb"] Oct 30 02:42:00.155702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:00.155663 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:00.539412 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:00.539370 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a29d403-91c3-4ab3-9f67-6d94fcfb557f" path="/var/lib/kubelet/pods/5a29d403-91c3-4ab3-9f67-6d94fcfb557f/volumes" Oct 30 02:42:01.404006 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:01.403968 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:01.521977 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:01.521939 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:01.842475 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:01.842383 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:01.968114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:01.956592 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:02.555731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:02.555695 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:03.456226 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:03.456187 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:04.697748 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:04.697709 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:04.805659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:04.805620 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:05.034292 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:05.034197 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:05.395463 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:05.395376 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:05.507047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:05.506958 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:06.662621 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:06.662573 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:06.768013 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:06.767973 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:08.241518 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:08.241479 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:09.641546 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:09.641508 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:09.969838 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:09.969802 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:10.369998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:10.369881 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:12.125865 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:12.125815 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:13.895219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:13.895166 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:15.118817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:15.118767 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:15.225924 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:15.225520 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:15.595772 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:15.595735 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:17.198675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.198637 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:17.482221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.482137 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x"] Oct 30 02:42:17.482714 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.482699 5998 cpu_manager.go:414] "containerMap: removing container" podUID="dd7460a2-c42d-4d5f-964e-525a49cdafff" containerName="istio-proxy" Oct 30 02:42:17.482765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.482716 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7460a2-c42d-4d5f-964e-525a49cdafff" containerName="istio-proxy" Oct 30 02:42:17.482765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.482738 5998 cpu_manager.go:414] "containerMap: removing container" podUID="5a29d403-91c3-4ab3-9f67-6d94fcfb557f" containerName="istio-proxy" Oct 30 02:42:17.482765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.482743 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a29d403-91c3-4ab3-9f67-6d94fcfb557f" containerName="istio-proxy" Oct 30 02:42:17.482765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.482756 5998 cpu_manager.go:414] "containerMap: removing container" podUID="5a29d403-91c3-4ab3-9f67-6d94fcfb557f" containerName="istio-proxy" Oct 30 02:42:17.482765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.482764 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a29d403-91c3-4ab3-9f67-6d94fcfb557f" containerName="istio-proxy" Oct 30 02:42:17.482929 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.482782 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8b5bea28-5420-4913-a4bb-eb8bbf18dbb7" containerName="revoke-keys" Oct 30 02:42:17.482929 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.482788 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b5bea28-5420-4913-a4bb-eb8bbf18dbb7" containerName="revoke-keys" Oct 30 02:42:17.482929 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.482882 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a29d403-91c3-4ab3-9f67-6d94fcfb557f" containerName="istio-proxy" Oct 30 02:42:17.482929 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.482892 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a29d403-91c3-4ab3-9f67-6d94fcfb557f" containerName="istio-proxy" Oct 30 02:42:17.482929 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.482898 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7460a2-c42d-4d5f-964e-525a49cdafff" containerName="istio-proxy" Oct 30 02:42:17.482929 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.482905 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b5bea28-5420-4913-a4bb-eb8bbf18dbb7" containerName="revoke-keys" Oct 30 02:42:17.486491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.486468 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x" Oct 30 02:42:17.501537 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.501507 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:42:17.506027 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.505993 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x"] Oct 30 02:42:17.512252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.512226 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:42:17.512421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.512369 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:42:17.512627 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.512612 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:42:17.616500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.616441 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgr7d\" (UniqueName: \"kubernetes.io/projected/15a69048-bb38-4e8d-9aed-012cc287ca39-kube-api-access-rgr7d\") pod \"maas-api-revoke-sub-867b366a-2sk6x\" (UID: \"15a69048-bb38-4e8d-9aed-012cc287ca39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x" Oct 30 02:42:17.616723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.616672 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15a69048-bb38-4e8d-9aed-012cc287ca39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-867b366a-2sk6x\" (UID: \"15a69048-bb38-4e8d-9aed-012cc287ca39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x" Oct 30 02:42:17.616804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.616720 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15a69048-bb38-4e8d-9aed-012cc287ca39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-867b366a-2sk6x\" (UID: \"15a69048-bb38-4e8d-9aed-012cc287ca39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x" Oct 30 02:42:17.718456 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.718414 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgr7d\" (UniqueName: \"kubernetes.io/projected/15a69048-bb38-4e8d-9aed-012cc287ca39-kube-api-access-rgr7d\") pod \"maas-api-revoke-sub-867b366a-2sk6x\" (UID: \"15a69048-bb38-4e8d-9aed-012cc287ca39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x" Oct 30 02:42:17.718671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.718525 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15a69048-bb38-4e8d-9aed-012cc287ca39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-867b366a-2sk6x\" (UID: \"15a69048-bb38-4e8d-9aed-012cc287ca39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x" Oct 30 02:42:17.718671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.718545 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15a69048-bb38-4e8d-9aed-012cc287ca39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-867b366a-2sk6x\" (UID: \"15a69048-bb38-4e8d-9aed-012cc287ca39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x" Oct 30 02:42:17.719482 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.719449 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15a69048-bb38-4e8d-9aed-012cc287ca39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-867b366a-2sk6x\" (UID: \"15a69048-bb38-4e8d-9aed-012cc287ca39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x" Oct 30 02:42:17.721167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.721136 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15a69048-bb38-4e8d-9aed-012cc287ca39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-867b366a-2sk6x\" (UID: \"15a69048-bb38-4e8d-9aed-012cc287ca39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x" Oct 30 02:42:17.740757 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.740668 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgr7d\" (UniqueName: \"kubernetes.io/projected/15a69048-bb38-4e8d-9aed-012cc287ca39-kube-api-access-rgr7d\") pod \"maas-api-revoke-sub-867b366a-2sk6x\" (UID: \"15a69048-bb38-4e8d-9aed-012cc287ca39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x" Oct 30 02:42:17.800781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:17.800740 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x" Oct 30 02:42:18.110265 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:18.110222 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x"] Oct 30 02:42:18.113785 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:42:18.113739 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a69048_bb38_4e8d_9aed_012cc287ca39.slice/crio-5d13677c2bade0bf8098b89d2f82f2812b7a0ebe9164e59c243505184769f481 WatchSource:0}: Error finding container 5d13677c2bade0bf8098b89d2f82f2812b7a0ebe9164e59c243505184769f481: Status 404 returned error can't find the container with id 5d13677c2bade0bf8098b89d2f82f2812b7a0ebe9164e59c243505184769f481 Oct 30 02:42:18.282150 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:18.282080 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x" event={"ID":"15a69048-bb38-4e8d-9aed-012cc287ca39","Type":"ContainerStarted","Data":"5d13677c2bade0bf8098b89d2f82f2812b7a0ebe9164e59c243505184769f481"} Oct 30 02:42:18.771342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:18.771255 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:19.290747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:19.290709 5998 generic.go:356] "Generic (PLEG): container finished" podID="15a69048-bb38-4e8d-9aed-012cc287ca39" containerID="6e93360f6ca39c0250c5a671d2e23c545e2458912927e602c963bc8a7ae78669" exitCode=0 Oct 30 02:42:19.291223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:19.290799 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x" event={"ID":"15a69048-bb38-4e8d-9aed-012cc287ca39","Type":"ContainerDied","Data":"6e93360f6ca39c0250c5a671d2e23c545e2458912927e602c963bc8a7ae78669"} Oct 30 02:42:20.099012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:20.098975 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:20.215116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:20.212829 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:20.784484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:20.784451 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x" Oct 30 02:42:20.901569 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:20.901525 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15a69048-bb38-4e8d-9aed-012cc287ca39-kube-api-access-rgr7d\" (UniqueName: \"kubernetes.io/projected/15a69048-bb38-4e8d-9aed-012cc287ca39-kube-api-access-rgr7d\") pod \"15a69048-bb38-4e8d-9aed-012cc287ca39\" (UID: \"15a69048-bb38-4e8d-9aed-012cc287ca39\") " Oct 30 02:42:20.901785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:20.901669 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15a69048-bb38-4e8d-9aed-012cc287ca39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15a69048-bb38-4e8d-9aed-012cc287ca39-maas-api-openshift-ca\") pod \"15a69048-bb38-4e8d-9aed-012cc287ca39\" (UID: \"15a69048-bb38-4e8d-9aed-012cc287ca39\") " Oct 30 02:42:20.901856 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:20.901780 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15a69048-bb38-4e8d-9aed-012cc287ca39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15a69048-bb38-4e8d-9aed-012cc287ca39-maas-api-xks-ca\") pod \"15a69048-bb38-4e8d-9aed-012cc287ca39\" (UID: \"15a69048-bb38-4e8d-9aed-012cc287ca39\") " Oct 30 02:42:20.902054 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:20.902023 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15a69048-bb38-4e8d-9aed-012cc287ca39-maas-api-openshift-ca" pod "15a69048-bb38-4e8d-9aed-012cc287ca39" (UID: "15a69048-bb38-4e8d-9aed-012cc287ca39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:42:20.902302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:20.902281 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15a69048-bb38-4e8d-9aed-012cc287ca39-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:42:20.903971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:20.903942 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15a69048-bb38-4e8d-9aed-012cc287ca39-maas-api-xks-ca" pod "15a69048-bb38-4e8d-9aed-012cc287ca39" (UID: "15a69048-bb38-4e8d-9aed-012cc287ca39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:42:20.904273 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:20.904220 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15a69048-bb38-4e8d-9aed-012cc287ca39-kube-api-access-rgr7d" pod "15a69048-bb38-4e8d-9aed-012cc287ca39" (UID: "15a69048-bb38-4e8d-9aed-012cc287ca39"). InnerVolumeSpecName "kube-api-access-rgr7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:42:20.984870 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:20.984779 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:21.003459 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:21.003418 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15a69048-bb38-4e8d-9aed-012cc287ca39-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:42:21.003459 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:21.003456 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgr7d\" (UniqueName: \"kubernetes.io/projected/15a69048-bb38-4e8d-9aed-012cc287ca39-kube-api-access-rgr7d\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:42:21.099153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:21.099053 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:21.359892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:21.359780 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x" event={"ID":"15a69048-bb38-4e8d-9aed-012cc287ca39","Type":"ContainerDied","Data":"5d13677c2bade0bf8098b89d2f82f2812b7a0ebe9164e59c243505184769f481"} Oct 30 02:42:21.359892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:21.359888 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d13677c2bade0bf8098b89d2f82f2812b7a0ebe9164e59c243505184769f481" Oct 30 02:42:21.360262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:21.359966 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x" Oct 30 02:42:21.397144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:21.395302 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:22.750250 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:22.750215 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:24.286789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:24.286741 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:25.222923 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:42:25.222785 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:42:25.871528 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:25.871493 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:26.458493 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:26.458463 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:26.695292 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:26.695251 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:27.155593 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:27.155554 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:27.549593 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:27.549560 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:27.820062 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:27.819937 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:27.997787 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:27.997746 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5"] Oct 30 02:42:27.998346 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:27.998329 5998 cpu_manager.go:414] "containerMap: removing container" podUID="15a69048-bb38-4e8d-9aed-012cc287ca39" containerName="revoke-keys" Oct 30 02:42:27.998346 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:27.998345 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15a69048-bb38-4e8d-9aed-012cc287ca39" containerName="revoke-keys" Oct 30 02:42:27.998534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:27.998462 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="15a69048-bb38-4e8d-9aed-012cc287ca39" containerName="revoke-keys" Oct 30 02:42:28.001946 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:28.001922 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5" Oct 30 02:42:28.025370 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:28.025342 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:42:28.025567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:28.025370 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:42:28.025689 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:28.025407 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:42:28.025764 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:28.025448 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:42:28.029336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:28.029312 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5"] Oct 30 02:42:28.139138 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:28.139034 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5664b661-de95-4e5a-bffd-84a09455f58c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d07dacc7-fjjf5\" (UID: \"5664b661-de95-4e5a-bffd-84a09455f58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5" Oct 30 02:42:28.139330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:28.139178 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5l7s\" (UniqueName: \"kubernetes.io/projected/5664b661-de95-4e5a-bffd-84a09455f58c-kube-api-access-b5l7s\") pod \"maas-api-revoke-sub-d07dacc7-fjjf5\" (UID: \"5664b661-de95-4e5a-bffd-84a09455f58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5" Oct 30 02:42:28.139330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:28.139202 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5664b661-de95-4e5a-bffd-84a09455f58c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d07dacc7-fjjf5\" (UID: \"5664b661-de95-4e5a-bffd-84a09455f58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5" Oct 30 02:42:28.240369 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:28.240331 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5664b661-de95-4e5a-bffd-84a09455f58c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d07dacc7-fjjf5\" (UID: \"5664b661-de95-4e5a-bffd-84a09455f58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5" Oct 30 02:42:28.240875 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:28.240465 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5l7s\" (UniqueName: \"kubernetes.io/projected/5664b661-de95-4e5a-bffd-84a09455f58c-kube-api-access-b5l7s\") pod \"maas-api-revoke-sub-d07dacc7-fjjf5\" (UID: \"5664b661-de95-4e5a-bffd-84a09455f58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5" Oct 30 02:42:28.240875 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:28.240501 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5664b661-de95-4e5a-bffd-84a09455f58c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d07dacc7-fjjf5\" (UID: \"5664b661-de95-4e5a-bffd-84a09455f58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5" Oct 30 02:42:28.241260 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:28.241236 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5664b661-de95-4e5a-bffd-84a09455f58c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d07dacc7-fjjf5\" (UID: \"5664b661-de95-4e5a-bffd-84a09455f58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5" Oct 30 02:42:28.242781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:28.242760 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5664b661-de95-4e5a-bffd-84a09455f58c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d07dacc7-fjjf5\" (UID: \"5664b661-de95-4e5a-bffd-84a09455f58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5" Oct 30 02:42:28.268549 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:28.268521 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5l7s\" (UniqueName: \"kubernetes.io/projected/5664b661-de95-4e5a-bffd-84a09455f58c-kube-api-access-b5l7s\") pod \"maas-api-revoke-sub-d07dacc7-fjjf5\" (UID: \"5664b661-de95-4e5a-bffd-84a09455f58c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5" Oct 30 02:42:28.316294 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:28.316254 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5" Oct 30 02:42:28.348302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:28.348269 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:28.628037 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:28.627982 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5"] Oct 30 02:42:28.631417 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:42:28.631390 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5664b661_de95_4e5a_bffd_84a09455f58c.slice/crio-2cbcf693e2907866e786760abd073b1a98200918292031305acbac2fcc5e24bd WatchSource:0}: Error finding container 2cbcf693e2907866e786760abd073b1a98200918292031305acbac2fcc5e24bd: Status 404 returned error can't find the container with id 2cbcf693e2907866e786760abd073b1a98200918292031305acbac2fcc5e24bd Oct 30 02:42:29.423962 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:29.423924 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5" event={"ID":"5664b661-de95-4e5a-bffd-84a09455f58c","Type":"ContainerStarted","Data":"af3b9dce154db9e5d8c45d9592c427e087e5391a2f8148a3162d6c1be89cb323"} Oct 30 02:42:29.424424 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:29.423976 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5" event={"ID":"5664b661-de95-4e5a-bffd-84a09455f58c","Type":"ContainerStarted","Data":"2cbcf693e2907866e786760abd073b1a98200918292031305acbac2fcc5e24bd"} Oct 30 02:42:29.462939 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:29.462872 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5" podStartSLOduration=2.462853629 podStartE2EDuration="2.462853629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:42:29.460812 +0000 UTC m=+4379.039603081" watchObservedRunningTime="2026-10-30 02:42:29.462853629 +0000 UTC m=+4379.041644710" Oct 30 02:42:29.534668 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:29.534630 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:30.434342 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:30.434303 5998 generic.go:356] "Generic (PLEG): container finished" podID="5664b661-de95-4e5a-bffd-84a09455f58c" containerID="af3b9dce154db9e5d8c45d9592c427e087e5391a2f8148a3162d6c1be89cb323" exitCode=0 Oct 30 02:42:30.434887 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:30.434382 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5" event={"ID":"5664b661-de95-4e5a-bffd-84a09455f58c","Type":"ContainerDied","Data":"af3b9dce154db9e5d8c45d9592c427e087e5391a2f8148a3162d6c1be89cb323"} Oct 30 02:42:31.041481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:31.041438 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:31.275058 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:31.274948 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:31.732541 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:31.732509 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5" Oct 30 02:42:31.791039 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:31.791002 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5664b661-de95-4e5a-bffd-84a09455f58c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5664b661-de95-4e5a-bffd-84a09455f58c-maas-api-xks-ca\") pod \"5664b661-de95-4e5a-bffd-84a09455f58c\" (UID: \"5664b661-de95-4e5a-bffd-84a09455f58c\") " Oct 30 02:42:31.791291 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:31.791066 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5664b661-de95-4e5a-bffd-84a09455f58c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5664b661-de95-4e5a-bffd-84a09455f58c-maas-api-openshift-ca\") pod \"5664b661-de95-4e5a-bffd-84a09455f58c\" (UID: \"5664b661-de95-4e5a-bffd-84a09455f58c\") " Oct 30 02:42:31.791291 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:31.791228 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5664b661-de95-4e5a-bffd-84a09455f58c-kube-api-access-b5l7s\" (UniqueName: \"kubernetes.io/projected/5664b661-de95-4e5a-bffd-84a09455f58c-kube-api-access-b5l7s\") pod \"5664b661-de95-4e5a-bffd-84a09455f58c\" (UID: \"5664b661-de95-4e5a-bffd-84a09455f58c\") " Oct 30 02:42:31.791486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:31.791455 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5664b661-de95-4e5a-bffd-84a09455f58c-maas-api-openshift-ca" pod "5664b661-de95-4e5a-bffd-84a09455f58c" (UID: "5664b661-de95-4e5a-bffd-84a09455f58c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:42:31.791789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:31.791769 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5664b661-de95-4e5a-bffd-84a09455f58c-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:42:31.793387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:31.793361 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5664b661-de95-4e5a-bffd-84a09455f58c-kube-api-access-b5l7s" pod "5664b661-de95-4e5a-bffd-84a09455f58c" (UID: "5664b661-de95-4e5a-bffd-84a09455f58c"). InnerVolumeSpecName "kube-api-access-b5l7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:42:31.793685 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:31.793668 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5664b661-de95-4e5a-bffd-84a09455f58c-maas-api-xks-ca" pod "5664b661-de95-4e5a-bffd-84a09455f58c" (UID: "5664b661-de95-4e5a-bffd-84a09455f58c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:42:31.892987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:31.892890 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5664b661-de95-4e5a-bffd-84a09455f58c-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:42:31.892987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:31.892933 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5l7s\" (UniqueName: \"kubernetes.io/projected/5664b661-de95-4e5a-bffd-84a09455f58c-kube-api-access-b5l7s\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:42:32.465877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:32.465833 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5" Oct 30 02:42:32.465877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:32.465856 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5" event={"ID":"5664b661-de95-4e5a-bffd-84a09455f58c","Type":"ContainerDied","Data":"2cbcf693e2907866e786760abd073b1a98200918292031305acbac2fcc5e24bd"} Oct 30 02:42:32.466183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:32.465919 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cbcf693e2907866e786760abd073b1a98200918292031305acbac2fcc5e24bd" Oct 30 02:42:32.863168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:32.861422 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:33.388976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:33.388933 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b"] Oct 30 02:42:33.542603 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:33.542036 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:33.650649 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:33.650510 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:34.694896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:34.694862 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:34.755274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:34.755237 5998 scope.go:122] "RemoveContainer" containerID="83a8694141a8d221fd73bbedce7176445c1fa62e41e25313d8b6e683d4d3b677" Oct 30 02:42:35.570473 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:35.570431 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:36.386913 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:36.386875 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:37.966688 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:37.966645 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:38.389735 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.389631 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b" podUID="85dfbbf9-122b-49ed-abab-a9c107c457ab" containerName="payload-processing" containerID="cri-o://3c9155f55f4e3282f259398678f1f6cc1526844abafd39914f964efb6a0998a4" gracePeriod=25 Oct 30 02:42:38.518778 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.518739 5998 generic.go:356] "Generic (PLEG): container finished" podID="85dfbbf9-122b-49ed-abab-a9c107c457ab" containerID="3c9155f55f4e3282f259398678f1f6cc1526844abafd39914f964efb6a0998a4" exitCode=0 Oct 30 02:42:38.518954 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.518807 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b" event={"ID":"85dfbbf9-122b-49ed-abab-a9c107c457ab","Type":"ContainerDied","Data":"3c9155f55f4e3282f259398678f1f6cc1526844abafd39914f964efb6a0998a4"} Oct 30 02:42:38.674781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.674753 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b" Oct 30 02:42:38.788422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.788384 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85dfbbf9-122b-49ed-abab-a9c107c457ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dfbbf9-122b-49ed-abab-a9c107c457ab-plugins-config-volume\") pod \"85dfbbf9-122b-49ed-abab-a9c107c457ab\" (UID: \"85dfbbf9-122b-49ed-abab-a9c107c457ab\") " Oct 30 02:42:38.788636 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.788596 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85dfbbf9-122b-49ed-abab-a9c107c457ab-kube-api-access-hzfpr\" (UniqueName: \"kubernetes.io/projected/85dfbbf9-122b-49ed-abab-a9c107c457ab-kube-api-access-hzfpr\") pod \"85dfbbf9-122b-49ed-abab-a9c107c457ab\" (UID: \"85dfbbf9-122b-49ed-abab-a9c107c457ab\") " Oct 30 02:42:38.788815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.788787 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85dfbbf9-122b-49ed-abab-a9c107c457ab-plugins-config-volume" pod "85dfbbf9-122b-49ed-abab-a9c107c457ab" (UID: "85dfbbf9-122b-49ed-abab-a9c107c457ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:42:38.789202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.789180 5998 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85dfbbf9-122b-49ed-abab-a9c107c457ab-plugins-config-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:42:38.790841 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.790816 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85dfbbf9-122b-49ed-abab-a9c107c457ab-kube-api-access-hzfpr" pod "85dfbbf9-122b-49ed-abab-a9c107c457ab" (UID: "85dfbbf9-122b-49ed-abab-a9c107c457ab"). InnerVolumeSpecName "kube-api-access-hzfpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:42:38.871954 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.871910 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv"] Oct 30 02:42:38.872553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.872531 5998 cpu_manager.go:414] "containerMap: removing container" podUID="5664b661-de95-4e5a-bffd-84a09455f58c" containerName="revoke-keys" Oct 30 02:42:38.872657 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.872555 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5664b661-de95-4e5a-bffd-84a09455f58c" containerName="revoke-keys" Oct 30 02:42:38.872657 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.872590 5998 cpu_manager.go:414] "containerMap: removing container" podUID="85dfbbf9-122b-49ed-abab-a9c107c457ab" containerName="payload-processing" Oct 30 02:42:38.872657 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.872599 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85dfbbf9-122b-49ed-abab-a9c107c457ab" containerName="payload-processing" Oct 30 02:42:38.872836 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.872796 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="5664b661-de95-4e5a-bffd-84a09455f58c" containerName="revoke-keys" Oct 30 02:42:38.872836 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.872818 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="85dfbbf9-122b-49ed-abab-a9c107c457ab" containerName="payload-processing" Oct 30 02:42:38.876312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.876290 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv" Oct 30 02:42:38.889498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.889448 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:42:38.889498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.889452 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:42:38.889661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.889540 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:42:38.889661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.889620 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:42:38.890177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.890158 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzfpr\" (UniqueName: \"kubernetes.io/projected/85dfbbf9-122b-49ed-abab-a9c107c457ab-kube-api-access-hzfpr\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:42:38.897218 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.897188 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv"] Oct 30 02:42:38.991737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.991700 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b3112b-1f2d-480f-bf87-7a5fe060f999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157dcde6-sztvv\" (UID: \"a8b3112b-1f2d-480f-bf87-7a5fe060f999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv" Oct 30 02:42:38.992181 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.991765 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf7z2\" (UniqueName: \"kubernetes.io/projected/a8b3112b-1f2d-480f-bf87-7a5fe060f999-kube-api-access-lf7z2\") pod \"maas-api-revoke-sub-157dcde6-sztvv\" (UID: \"a8b3112b-1f2d-480f-bf87-7a5fe060f999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv" Oct 30 02:42:38.992181 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:38.991876 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b3112b-1f2d-480f-bf87-7a5fe060f999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157dcde6-sztvv\" (UID: \"a8b3112b-1f2d-480f-bf87-7a5fe060f999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv" Oct 30 02:42:39.092986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:39.092945 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b3112b-1f2d-480f-bf87-7a5fe060f999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157dcde6-sztvv\" (UID: \"a8b3112b-1f2d-480f-bf87-7a5fe060f999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv" Oct 30 02:42:39.093212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:39.093009 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf7z2\" (UniqueName: \"kubernetes.io/projected/a8b3112b-1f2d-480f-bf87-7a5fe060f999-kube-api-access-lf7z2\") pod \"maas-api-revoke-sub-157dcde6-sztvv\" (UID: \"a8b3112b-1f2d-480f-bf87-7a5fe060f999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv" Oct 30 02:42:39.093212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:39.093068 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b3112b-1f2d-480f-bf87-7a5fe060f999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157dcde6-sztvv\" (UID: \"a8b3112b-1f2d-480f-bf87-7a5fe060f999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv" Oct 30 02:42:39.093755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:39.093733 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b3112b-1f2d-480f-bf87-7a5fe060f999-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157dcde6-sztvv\" (UID: \"a8b3112b-1f2d-480f-bf87-7a5fe060f999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv" Oct 30 02:42:39.095384 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:39.095365 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b3112b-1f2d-480f-bf87-7a5fe060f999-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157dcde6-sztvv\" (UID: \"a8b3112b-1f2d-480f-bf87-7a5fe060f999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv" Oct 30 02:42:39.115703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:39.115659 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf7z2\" (UniqueName: \"kubernetes.io/projected/a8b3112b-1f2d-480f-bf87-7a5fe060f999-kube-api-access-lf7z2\") pod \"maas-api-revoke-sub-157dcde6-sztvv\" (UID: \"a8b3112b-1f2d-480f-bf87-7a5fe060f999\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv" Oct 30 02:42:39.190639 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:39.190603 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv" Oct 30 02:42:39.502466 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:39.502434 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv"] Oct 30 02:42:39.505036 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:42:39.504573 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b3112b_1f2d_480f_bf87_7a5fe060f999.slice/crio-6d0160e805d354d377dba2e0d129f3309ca11ffbaea40b1e5f6e6443b2c03b57 WatchSource:0}: Error finding container 6d0160e805d354d377dba2e0d129f3309ca11ffbaea40b1e5f6e6443b2c03b57: Status 404 returned error can't find the container with id 6d0160e805d354d377dba2e0d129f3309ca11ffbaea40b1e5f6e6443b2c03b57 Oct 30 02:42:39.530025 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:39.530000 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b" Oct 30 02:42:39.530181 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:39.530021 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b" event={"ID":"85dfbbf9-122b-49ed-abab-a9c107c457ab","Type":"ContainerDied","Data":"0d5b00583ed1f9d92aad1247a200faf2d0a948119a09cf6d1e0936278f4d7dc2"} Oct 30 02:42:39.530181 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:39.530110 5998 scope.go:122] "RemoveContainer" containerID="3c9155f55f4e3282f259398678f1f6cc1526844abafd39914f964efb6a0998a4" Oct 30 02:42:39.533552 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:39.533525 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv" event={"ID":"a8b3112b-1f2d-480f-bf87-7a5fe060f999","Type":"ContainerStarted","Data":"6d0160e805d354d377dba2e0d129f3309ca11ffbaea40b1e5f6e6443b2c03b57"} Oct 30 02:42:39.575337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:39.575306 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:39.583225 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:39.583195 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b"] Oct 30 02:42:39.587256 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:39.587225 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a0684f-5cc6fbf578-pj79b"] Oct 30 02:42:40.542354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:40.542306 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85dfbbf9-122b-49ed-abab-a9c107c457ab" path="/var/lib/kubelet/pods/85dfbbf9-122b-49ed-abab-a9c107c457ab/volumes" Oct 30 02:42:40.544265 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:40.544230 5998 generic.go:356] "Generic (PLEG): container finished" podID="a8b3112b-1f2d-480f-bf87-7a5fe060f999" containerID="db3d735fe337c4e9e99799dc258699178ac77443991b489d26ab2df7d51b745c" exitCode=0 Oct 30 02:42:40.544396 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:40.544304 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv" event={"ID":"a8b3112b-1f2d-480f-bf87-7a5fe060f999","Type":"ContainerDied","Data":"db3d735fe337c4e9e99799dc258699178ac77443991b489d26ab2df7d51b745c"} Oct 30 02:42:41.102762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:41.102725 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:41.210682 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:41.210648 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:41.708135 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:41.708080 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv" Oct 30 02:42:41.837506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:41.837398 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b3112b-1f2d-480f-bf87-7a5fe060f999-kube-api-access-lf7z2\" (UniqueName: \"kubernetes.io/projected/a8b3112b-1f2d-480f-bf87-7a5fe060f999-kube-api-access-lf7z2\") pod \"a8b3112b-1f2d-480f-bf87-7a5fe060f999\" (UID: \"a8b3112b-1f2d-480f-bf87-7a5fe060f999\") " Oct 30 02:42:41.837506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:41.837467 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b3112b-1f2d-480f-bf87-7a5fe060f999-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b3112b-1f2d-480f-bf87-7a5fe060f999-maas-api-xks-ca\") pod \"a8b3112b-1f2d-480f-bf87-7a5fe060f999\" (UID: \"a8b3112b-1f2d-480f-bf87-7a5fe060f999\") " Oct 30 02:42:41.837764 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:41.837674 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b3112b-1f2d-480f-bf87-7a5fe060f999-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b3112b-1f2d-480f-bf87-7a5fe060f999-maas-api-openshift-ca\") pod \"a8b3112b-1f2d-480f-bf87-7a5fe060f999\" (UID: \"a8b3112b-1f2d-480f-bf87-7a5fe060f999\") " Oct 30 02:42:41.838156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:41.838127 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b3112b-1f2d-480f-bf87-7a5fe060f999-maas-api-openshift-ca" pod "a8b3112b-1f2d-480f-bf87-7a5fe060f999" (UID: "a8b3112b-1f2d-480f-bf87-7a5fe060f999"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:42:41.839760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:41.839733 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b3112b-1f2d-480f-bf87-7a5fe060f999-maas-api-xks-ca" pod "a8b3112b-1f2d-480f-bf87-7a5fe060f999" (UID: "a8b3112b-1f2d-480f-bf87-7a5fe060f999"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:42:41.839871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:41.839806 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b3112b-1f2d-480f-bf87-7a5fe060f999-kube-api-access-lf7z2" pod "a8b3112b-1f2d-480f-bf87-7a5fe060f999" (UID: "a8b3112b-1f2d-480f-bf87-7a5fe060f999"). InnerVolumeSpecName "kube-api-access-lf7z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:42:41.939789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:41.939742 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8b3112b-1f2d-480f-bf87-7a5fe060f999-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:42:41.939789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:41.939780 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf7z2\" (UniqueName: \"kubernetes.io/projected/a8b3112b-1f2d-480f-bf87-7a5fe060f999-kube-api-access-lf7z2\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:42:41.939789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:41.939792 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8b3112b-1f2d-480f-bf87-7a5fe060f999-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:42:42.249021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:42.248982 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:42.561458 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:42.561424 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv" Oct 30 02:42:42.561672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:42.561434 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv" event={"ID":"a8b3112b-1f2d-480f-bf87-7a5fe060f999","Type":"ContainerDied","Data":"6d0160e805d354d377dba2e0d129f3309ca11ffbaea40b1e5f6e6443b2c03b57"} Oct 30 02:42:42.561672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:42.561596 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d0160e805d354d377dba2e0d129f3309ca11ffbaea40b1e5f6e6443b2c03b57" Oct 30 02:42:42.565130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:42.565063 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:43.036767 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:43.036730 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf"] Oct 30 02:42:43.044695 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:43.044664 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17631d18-zx9xf"] Oct 30 02:42:44.168021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:44.167981 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:44.542393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:44.542281 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977" path="/var/lib/kubelet/pods/1b53f6b8-8a57-4f6b-b4fe-5a13da5a6977/volumes" Oct 30 02:42:45.651906 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:45.651865 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:46.598010 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:46.597972 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:42:47.489046 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:47.489009 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:49.375994 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.375959 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:49.532672 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.532634 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54"] Oct 30 02:42:49.533282 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.533258 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a8b3112b-1f2d-480f-bf87-7a5fe060f999" containerName="revoke-keys" Oct 30 02:42:49.533282 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.533282 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b3112b-1f2d-480f-bf87-7a5fe060f999" containerName="revoke-keys" Oct 30 02:42:49.533478 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.533438 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b3112b-1f2d-480f-bf87-7a5fe060f999" containerName="revoke-keys" Oct 30 02:42:49.538118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.538083 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54" Oct 30 02:42:49.565518 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.565477 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:42:49.565693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.565502 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:42:49.565783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.565522 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:42:49.565783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.565540 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:42:49.574976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.574940 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54"] Oct 30 02:42:49.660423 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.660333 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb180c8-0839-4632-8fe3-613d4e5afb82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e3c8990-tnr54\" (UID: \"7eb180c8-0839-4632-8fe3-613d4e5afb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54" Oct 30 02:42:49.660563 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.660530 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dz6n\" (UniqueName: \"kubernetes.io/projected/7eb180c8-0839-4632-8fe3-613d4e5afb82-kube-api-access-4dz6n\") pod \"maas-api-revoke-sub-6e3c8990-tnr54\" (UID: \"7eb180c8-0839-4632-8fe3-613d4e5afb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54" Oct 30 02:42:49.660674 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.660655 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb180c8-0839-4632-8fe3-613d4e5afb82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e3c8990-tnr54\" (UID: \"7eb180c8-0839-4632-8fe3-613d4e5afb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54" Oct 30 02:42:49.762435 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.762388 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb180c8-0839-4632-8fe3-613d4e5afb82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e3c8990-tnr54\" (UID: \"7eb180c8-0839-4632-8fe3-613d4e5afb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54" Oct 30 02:42:49.762617 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.762459 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb180c8-0839-4632-8fe3-613d4e5afb82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e3c8990-tnr54\" (UID: \"7eb180c8-0839-4632-8fe3-613d4e5afb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54" Oct 30 02:42:49.762705 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.762629 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dz6n\" (UniqueName: \"kubernetes.io/projected/7eb180c8-0839-4632-8fe3-613d4e5afb82-kube-api-access-4dz6n\") pod \"maas-api-revoke-sub-6e3c8990-tnr54\" (UID: \"7eb180c8-0839-4632-8fe3-613d4e5afb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54" Oct 30 02:42:49.763083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.763054 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb180c8-0839-4632-8fe3-613d4e5afb82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e3c8990-tnr54\" (UID: \"7eb180c8-0839-4632-8fe3-613d4e5afb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54" Oct 30 02:42:49.764956 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.764927 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb180c8-0839-4632-8fe3-613d4e5afb82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e3c8990-tnr54\" (UID: \"7eb180c8-0839-4632-8fe3-613d4e5afb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54" Oct 30 02:42:49.784567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.784535 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dz6n\" (UniqueName: \"kubernetes.io/projected/7eb180c8-0839-4632-8fe3-613d4e5afb82-kube-api-access-4dz6n\") pod \"maas-api-revoke-sub-6e3c8990-tnr54\" (UID: \"7eb180c8-0839-4632-8fe3-613d4e5afb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54" Oct 30 02:42:49.851718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:49.851680 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54" Oct 30 02:42:50.157973 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:50.157930 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54"] Oct 30 02:42:50.161498 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:42:50.161467 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb180c8_0839_4632_8fe3_613d4e5afb82.slice/crio-d67fcf6a523d16f2f9c3d2607be7d46068c385fc38ceb7e5ecd1334b0a44f4e8 WatchSource:0}: Error finding container d67fcf6a523d16f2f9c3d2607be7d46068c385fc38ceb7e5ecd1334b0a44f4e8: Status 404 returned error can't find the container with id d67fcf6a523d16f2f9c3d2607be7d46068c385fc38ceb7e5ecd1334b0a44f4e8 Oct 30 02:42:50.635583 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:50.635543 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54" event={"ID":"7eb180c8-0839-4632-8fe3-613d4e5afb82","Type":"ContainerStarted","Data":"e62fcdb35e6ad35b0e3ac5e5eae4cb38d6b47bd08f557ef84e2160690c48f344"} Oct 30 02:42:50.636011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:50.635607 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54" event={"ID":"7eb180c8-0839-4632-8fe3-613d4e5afb82","Type":"ContainerStarted","Data":"d67fcf6a523d16f2f9c3d2607be7d46068c385fc38ceb7e5ecd1334b0a44f4e8"} Oct 30 02:42:50.676632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:50.676573 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54" podStartSLOduration=1.676558523 podStartE2EDuration="1.676558523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:42:50.672290663 +0000 UTC m=+4400.251081744" watchObservedRunningTime="2026-10-30 02:42:50.676558523 +0000 UTC m=+4400.255349603" Oct 30 02:42:50.989754 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:50.989712 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:51.643692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:51.643653 5998 generic.go:356] "Generic (PLEG): container finished" podID="7eb180c8-0839-4632-8fe3-613d4e5afb82" containerID="e62fcdb35e6ad35b0e3ac5e5eae4cb38d6b47bd08f557ef84e2160690c48f344" exitCode=0 Oct 30 02:42:51.644160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:51.643744 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54" event={"ID":"7eb180c8-0839-4632-8fe3-613d4e5afb82","Type":"ContainerDied","Data":"e62fcdb35e6ad35b0e3ac5e5eae4cb38d6b47bd08f557ef84e2160690c48f344"} Oct 30 02:42:52.479302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:52.479261 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:52.809255 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:52.809221 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54" Oct 30 02:42:52.903700 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:52.903659 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eb180c8-0839-4632-8fe3-613d4e5afb82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb180c8-0839-4632-8fe3-613d4e5afb82-maas-api-xks-ca\") pod \"7eb180c8-0839-4632-8fe3-613d4e5afb82\" (UID: \"7eb180c8-0839-4632-8fe3-613d4e5afb82\") " Oct 30 02:42:52.903903 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:52.903728 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eb180c8-0839-4632-8fe3-613d4e5afb82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb180c8-0839-4632-8fe3-613d4e5afb82-maas-api-openshift-ca\") pod \"7eb180c8-0839-4632-8fe3-613d4e5afb82\" (UID: \"7eb180c8-0839-4632-8fe3-613d4e5afb82\") " Oct 30 02:42:52.903903 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:52.903854 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eb180c8-0839-4632-8fe3-613d4e5afb82-kube-api-access-4dz6n\" (UniqueName: \"kubernetes.io/projected/7eb180c8-0839-4632-8fe3-613d4e5afb82-kube-api-access-4dz6n\") pod \"7eb180c8-0839-4632-8fe3-613d4e5afb82\" (UID: \"7eb180c8-0839-4632-8fe3-613d4e5afb82\") " Oct 30 02:42:52.904228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:52.904203 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb180c8-0839-4632-8fe3-613d4e5afb82-maas-api-openshift-ca" pod "7eb180c8-0839-4632-8fe3-613d4e5afb82" (UID: "7eb180c8-0839-4632-8fe3-613d4e5afb82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:42:52.904480 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:52.904462 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb180c8-0839-4632-8fe3-613d4e5afb82-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:42:52.906693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:52.906664 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb180c8-0839-4632-8fe3-613d4e5afb82-kube-api-access-4dz6n" pod "7eb180c8-0839-4632-8fe3-613d4e5afb82" (UID: "7eb180c8-0839-4632-8fe3-613d4e5afb82"). InnerVolumeSpecName "kube-api-access-4dz6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:42:52.906798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:52.906720 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eb180c8-0839-4632-8fe3-613d4e5afb82-maas-api-xks-ca" pod "7eb180c8-0839-4632-8fe3-613d4e5afb82" (UID: "7eb180c8-0839-4632-8fe3-613d4e5afb82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:42:53.005428 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:53.005384 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dz6n\" (UniqueName: \"kubernetes.io/projected/7eb180c8-0839-4632-8fe3-613d4e5afb82-kube-api-access-4dz6n\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:42:53.005428 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:53.005421 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb180c8-0839-4632-8fe3-613d4e5afb82-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:42:53.659057 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:53.659020 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54" Oct 30 02:42:53.659281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:53.659058 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54" event={"ID":"7eb180c8-0839-4632-8fe3-613d4e5afb82","Type":"ContainerDied","Data":"d67fcf6a523d16f2f9c3d2607be7d46068c385fc38ceb7e5ecd1334b0a44f4e8"} Oct 30 02:42:53.659281 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:53.659149 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d67fcf6a523d16f2f9c3d2607be7d46068c385fc38ceb7e5ecd1334b0a44f4e8" Oct 30 02:42:54.056465 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:54.056403 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:55.031771 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:55.031728 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s"] Oct 30 02:42:55.044049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:55.044010 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd980138-lww2s"] Oct 30 02:42:55.222462 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:42:55.222423 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:42:55.586139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:55.586075 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:56.540916 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:56.540869 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c" path="/var/lib/kubelet/pods/7af8fdff-1df6-4cf2-9cc1-4a5306c9a14c/volumes" Oct 30 02:42:57.206838 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:57.206802 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:58.507676 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:58.507618 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:42:59.167008 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:42:59.166969 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:00.032883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:00.032845 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z"] Oct 30 02:43:00.042905 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:00.042872 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbb9d0b5-6gp8z"] Oct 30 02:43:00.540343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:00.540307 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30eb3269-4d9d-4423-ab4d-0b4ff17d2169" path="/var/lib/kubelet/pods/30eb3269-4d9d-4423-ab4d-0b4ff17d2169/volumes" Oct 30 02:43:00.576909 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:00.576876 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:00.689399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:00.689359 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:00.801943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:00.801862 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:01.325361 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:01.325321 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:01.652030 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:01.651926 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9"] Oct 30 02:43:01.652268 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:01.652237 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" podUID="563a9420-524b-48cc-9e77-2b6001604841" containerName="istio-proxy" containerID="cri-o://2606564cb9b531788a30350070a53a0830106b3756f2c6b14874100fe5779950" gracePeriod=30 Oct 30 02:43:01.696819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:01.696751 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:02.154631 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.154596 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:02.669084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.669039 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f"] Oct 30 02:43:02.669656 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.669634 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7eb180c8-0839-4632-8fe3-613d4e5afb82" containerName="revoke-keys" Oct 30 02:43:02.669708 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.669657 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eb180c8-0839-4632-8fe3-613d4e5afb82" containerName="revoke-keys" Oct 30 02:43:02.669792 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.669782 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eb180c8-0839-4632-8fe3-613d4e5afb82" containerName="revoke-keys" Oct 30 02:43:02.674517 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.674493 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f" Oct 30 02:43:02.690248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.690215 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:43:02.697150 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.697083 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:02.700584 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.700557 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:43:02.700813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.700617 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:43:02.700813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.700689 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:43:02.705881 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.705846 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f"] Oct 30 02:43:02.776419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.776380 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e8ed90-b3fb-4374-9380-733684ba2cb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22df452a-v577f\" (UID: \"c5e8ed90-b3fb-4374-9380-733684ba2cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f" Oct 30 02:43:02.776664 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.776641 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zs88\" (UniqueName: \"kubernetes.io/projected/c5e8ed90-b3fb-4374-9380-733684ba2cb9-kube-api-access-6zs88\") pod \"maas-api-revoke-sub-22df452a-v577f\" (UID: \"c5e8ed90-b3fb-4374-9380-733684ba2cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f" Oct 30 02:43:02.776788 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.776765 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e8ed90-b3fb-4374-9380-733684ba2cb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22df452a-v577f\" (UID: \"c5e8ed90-b3fb-4374-9380-733684ba2cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f" Oct 30 02:43:02.878405 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.878347 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zs88\" (UniqueName: \"kubernetes.io/projected/c5e8ed90-b3fb-4374-9380-733684ba2cb9-kube-api-access-6zs88\") pod \"maas-api-revoke-sub-22df452a-v577f\" (UID: \"c5e8ed90-b3fb-4374-9380-733684ba2cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f" Oct 30 02:43:02.878628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.878470 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e8ed90-b3fb-4374-9380-733684ba2cb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22df452a-v577f\" (UID: \"c5e8ed90-b3fb-4374-9380-733684ba2cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f" Oct 30 02:43:02.878628 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.878499 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e8ed90-b3fb-4374-9380-733684ba2cb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22df452a-v577f\" (UID: \"c5e8ed90-b3fb-4374-9380-733684ba2cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f" Oct 30 02:43:02.879235 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.879208 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e8ed90-b3fb-4374-9380-733684ba2cb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22df452a-v577f\" (UID: \"c5e8ed90-b3fb-4374-9380-733684ba2cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f" Oct 30 02:43:02.881046 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.881021 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e8ed90-b3fb-4374-9380-733684ba2cb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22df452a-v577f\" (UID: \"c5e8ed90-b3fb-4374-9380-733684ba2cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f" Oct 30 02:43:02.901117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.901071 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zs88\" (UniqueName: \"kubernetes.io/projected/c5e8ed90-b3fb-4374-9380-733684ba2cb9-kube-api-access-6zs88\") pod \"maas-api-revoke-sub-22df452a-v577f\" (UID: \"c5e8ed90-b3fb-4374-9380-733684ba2cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f" Oct 30 02:43:02.989063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:02.988968 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f" Oct 30 02:43:03.302054 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:03.302022 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f"] Oct 30 02:43:03.305356 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:43:03.305317 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e8ed90_b3fb_4374_9380_733684ba2cb9.slice/crio-a1df2ec82cfdf31b18f69c0127fa07f20aa5be47fbf3ac75dec0d662bcad6457 WatchSource:0}: Error finding container a1df2ec82cfdf31b18f69c0127fa07f20aa5be47fbf3ac75dec0d662bcad6457: Status 404 returned error can't find the container with id a1df2ec82cfdf31b18f69c0127fa07f20aa5be47fbf3ac75dec0d662bcad6457 Oct 30 02:43:03.827649 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:03.827605 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f" event={"ID":"c5e8ed90-b3fb-4374-9380-733684ba2cb9","Type":"ContainerStarted","Data":"fa26205384f46214855ec05ea7b115d532a46a1d70d97add399f28f44613f6c3"} Oct 30 02:43:03.828197 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:03.827677 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f" event={"ID":"c5e8ed90-b3fb-4374-9380-733684ba2cb9","Type":"ContainerStarted","Data":"a1df2ec82cfdf31b18f69c0127fa07f20aa5be47fbf3ac75dec0d662bcad6457"} Oct 30 02:43:03.868254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:03.868119 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f" podStartSLOduration=1.868084707 podStartE2EDuration="1.868084707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:43:03.865014311 +0000 UTC m=+4413.443805392" watchObservedRunningTime="2026-10-30 02:43:03.868084707 +0000 UTC m=+4413.446875787" Oct 30 02:43:04.292817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:04.292775 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:04.838278 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:04.838242 5998 generic.go:356] "Generic (PLEG): container finished" podID="c5e8ed90-b3fb-4374-9380-733684ba2cb9" containerID="fa26205384f46214855ec05ea7b115d532a46a1d70d97add399f28f44613f6c3" exitCode=0 Oct 30 02:43:04.838711 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:04.838328 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f" event={"ID":"c5e8ed90-b3fb-4374-9380-733684ba2cb9","Type":"ContainerDied","Data":"fa26205384f46214855ec05ea7b115d532a46a1d70d97add399f28f44613f6c3"} Oct 30 02:43:05.975146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:05.968901 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:06.052048 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.052023 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f" Oct 30 02:43:06.157910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.157820 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e8ed90-b3fb-4374-9380-733684ba2cb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e8ed90-b3fb-4374-9380-733684ba2cb9-maas-api-openshift-ca\") pod \"c5e8ed90-b3fb-4374-9380-733684ba2cb9\" (UID: \"c5e8ed90-b3fb-4374-9380-733684ba2cb9\") " Oct 30 02:43:06.157910 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.157892 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e8ed90-b3fb-4374-9380-733684ba2cb9-kube-api-access-6zs88\" (UniqueName: \"kubernetes.io/projected/c5e8ed90-b3fb-4374-9380-733684ba2cb9-kube-api-access-6zs88\") pod \"c5e8ed90-b3fb-4374-9380-733684ba2cb9\" (UID: \"c5e8ed90-b3fb-4374-9380-733684ba2cb9\") " Oct 30 02:43:06.158206 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.158052 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5e8ed90-b3fb-4374-9380-733684ba2cb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e8ed90-b3fb-4374-9380-733684ba2cb9-maas-api-xks-ca\") pod \"c5e8ed90-b3fb-4374-9380-733684ba2cb9\" (UID: \"c5e8ed90-b3fb-4374-9380-733684ba2cb9\") " Oct 30 02:43:06.158309 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.158284 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e8ed90-b3fb-4374-9380-733684ba2cb9-maas-api-openshift-ca" pod "c5e8ed90-b3fb-4374-9380-733684ba2cb9" (UID: "c5e8ed90-b3fb-4374-9380-733684ba2cb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:43:06.158582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.158561 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5e8ed90-b3fb-4374-9380-733684ba2cb9-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:06.160081 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.160055 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e8ed90-b3fb-4374-9380-733684ba2cb9-kube-api-access-6zs88" pod "c5e8ed90-b3fb-4374-9380-733684ba2cb9" (UID: "c5e8ed90-b3fb-4374-9380-733684ba2cb9"). InnerVolumeSpecName "kube-api-access-6zs88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:43:06.160223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.160114 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e8ed90-b3fb-4374-9380-733684ba2cb9-maas-api-xks-ca" pod "c5e8ed90-b3fb-4374-9380-733684ba2cb9" (UID: "c5e8ed90-b3fb-4374-9380-733684ba2cb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:43:06.259770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.259726 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zs88\" (UniqueName: \"kubernetes.io/projected/c5e8ed90-b3fb-4374-9380-733684ba2cb9-kube-api-access-6zs88\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:06.259770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.259768 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5e8ed90-b3fb-4374-9380-733684ba2cb9-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:06.418464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.418412 5998 patch_prober.go:29] interesting pod/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:43:06.418647 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.418510 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" podUID="563a9420-524b-48cc-9e77-2b6001604841" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:43:06.857890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.857855 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f" Oct 30 02:43:06.858107 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.857935 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f" event={"ID":"c5e8ed90-b3fb-4374-9380-733684ba2cb9","Type":"ContainerDied","Data":"a1df2ec82cfdf31b18f69c0127fa07f20aa5be47fbf3ac75dec0d662bcad6457"} Oct 30 02:43:06.858107 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.857976 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1df2ec82cfdf31b18f69c0127fa07f20aa5be47fbf3ac75dec0d662bcad6457" Oct 30 02:43:06.866387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.866050 5998 generic.go:356] "Generic (PLEG): container finished" podID="563a9420-524b-48cc-9e77-2b6001604841" containerID="2606564cb9b531788a30350070a53a0830106b3756f2c6b14874100fe5779950" exitCode=0 Oct 30 02:43:06.866387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.866228 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" event={"ID":"563a9420-524b-48cc-9e77-2b6001604841","Type":"ContainerDied","Data":"2606564cb9b531788a30350070a53a0830106b3756f2c6b14874100fe5779950"} Oct 30 02:43:06.866387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.866283 5998 scope.go:122] "RemoveContainer" containerID="2356216a486173038d8e3dc5102885229785d20bddb29cf0ee8de60ee884bc81" Oct 30 02:43:06.937029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.936957 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:43:06.969511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.969461 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-istio-data\") pod \"563a9420-524b-48cc-9e77-2b6001604841\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " Oct 30 02:43:06.969721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.969528 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/563a9420-524b-48cc-9e77-2b6001604841-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/563a9420-524b-48cc-9e77-2b6001604841-istiod-ca-cert\") pod \"563a9420-524b-48cc-9e77-2b6001604841\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " Oct 30 02:43:06.969721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.969583 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/563a9420-524b-48cc-9e77-2b6001604841-kube-api-access-9k94w\" (UniqueName: \"kubernetes.io/projected/563a9420-524b-48cc-9e77-2b6001604841-kube-api-access-9k94w\") pod \"563a9420-524b-48cc-9e77-2b6001604841\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " Oct 30 02:43:06.969721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.969626 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/563a9420-524b-48cc-9e77-2b6001604841-istio-token\" (UniqueName: \"kubernetes.io/projected/563a9420-524b-48cc-9e77-2b6001604841-istio-token\") pod \"563a9420-524b-48cc-9e77-2b6001604841\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " Oct 30 02:43:06.969721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.969681 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/563a9420-524b-48cc-9e77-2b6001604841-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/563a9420-524b-48cc-9e77-2b6001604841-istio-podinfo\") pod \"563a9420-524b-48cc-9e77-2b6001604841\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " Oct 30 02:43:06.969976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.969793 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-workload-socket\") pod \"563a9420-524b-48cc-9e77-2b6001604841\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " Oct 30 02:43:06.969976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.969796 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-istio-data" pod "563a9420-524b-48cc-9e77-2b6001604841" (UID: "563a9420-524b-48cc-9e77-2b6001604841"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:06.969976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.969876 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-credential-socket\") pod \"563a9420-524b-48cc-9e77-2b6001604841\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " Oct 30 02:43:06.969976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.969906 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-istio-envoy\") pod \"563a9420-524b-48cc-9e77-2b6001604841\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " Oct 30 02:43:06.969976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.969967 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-workload-certs\") pod \"563a9420-524b-48cc-9e77-2b6001604841\" (UID: \"563a9420-524b-48cc-9e77-2b6001604841\") " Oct 30 02:43:06.970280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.970020 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/563a9420-524b-48cc-9e77-2b6001604841-istiod-ca-cert" pod "563a9420-524b-48cc-9e77-2b6001604841" (UID: "563a9420-524b-48cc-9e77-2b6001604841"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:43:06.970346 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.970313 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-credential-socket" pod "563a9420-524b-48cc-9e77-2b6001604841" (UID: "563a9420-524b-48cc-9e77-2b6001604841"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:06.970461 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.970436 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-workload-socket" pod "563a9420-524b-48cc-9e77-2b6001604841" (UID: "563a9420-524b-48cc-9e77-2b6001604841"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:06.970721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.970690 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-workload-certs" pod "563a9420-524b-48cc-9e77-2b6001604841" (UID: "563a9420-524b-48cc-9e77-2b6001604841"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:06.970883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.970703 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:06.971043 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.971027 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/563a9420-524b-48cc-9e77-2b6001604841-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:06.971196 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.971179 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:06.971306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.971292 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:06.972427 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.972373 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563a9420-524b-48cc-9e77-2b6001604841-kube-api-access-9k94w" pod "563a9420-524b-48cc-9e77-2b6001604841" (UID: "563a9420-524b-48cc-9e77-2b6001604841"). InnerVolumeSpecName "kube-api-access-9k94w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:43:06.972732 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.972712 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563a9420-524b-48cc-9e77-2b6001604841-istio-token" pod "563a9420-524b-48cc-9e77-2b6001604841" (UID: "563a9420-524b-48cc-9e77-2b6001604841"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:43:06.972869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.972851 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/563a9420-524b-48cc-9e77-2b6001604841-istio-podinfo" pod "563a9420-524b-48cc-9e77-2b6001604841" (UID: "563a9420-524b-48cc-9e77-2b6001604841"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:43:06.972935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:06.972893 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-istio-envoy" pod "563a9420-524b-48cc-9e77-2b6001604841" (UID: "563a9420-524b-48cc-9e77-2b6001604841"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:07.072215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:07.072178 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k94w\" (UniqueName: \"kubernetes.io/projected/563a9420-524b-48cc-9e77-2b6001604841-kube-api-access-9k94w\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:07.072215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:07.072211 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/563a9420-524b-48cc-9e77-2b6001604841-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:07.072215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:07.072221 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/563a9420-524b-48cc-9e77-2b6001604841-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:07.072215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:07.072229 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:07.072728 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:07.072237 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/563a9420-524b-48cc-9e77-2b6001604841-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:07.124430 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:07.124391 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d"] Oct 30 02:43:07.131425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:07.131392 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7"] Oct 30 02:43:07.146055 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:07.146019 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv"] Oct 30 02:43:07.219829 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:07.219698 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:07.497732 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:07.497656 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:07.876300 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:07.876219 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" Oct 30 02:43:07.876468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:07.876212 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9" event={"ID":"563a9420-524b-48cc-9e77-2b6001604841","Type":"ContainerDied","Data":"6287334840b2171729c721739124398b2bc95568d888a440566cb0426724fd4f"} Oct 30 02:43:07.876468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:07.876382 5998 scope.go:122] "RemoveContainer" containerID="2606564cb9b531788a30350070a53a0830106b3756f2c6b14874100fe5779950" Oct 30 02:43:07.925476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:07.925434 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9"] Oct 30 02:43:07.935843 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:07.935806 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-a0684f-openshift-default-66d4745cfc-zq5d9"] Oct 30 02:43:07.950504 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:07.950468 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:08.540899 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:08.540863 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="563a9420-524b-48cc-9e77-2b6001604841" path="/var/lib/kubelet/pods/563a9420-524b-48cc-9e77-2b6001604841/volumes" Oct 30 02:43:09.073562 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:09.073527 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:10.640458 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:10.640422 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:12.277065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:12.276972 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:13.163266 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.163232 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx"] Oct 30 02:43:13.163773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.163759 5998 cpu_manager.go:414] "containerMap: removing container" podUID="563a9420-524b-48cc-9e77-2b6001604841" containerName="istio-proxy" Oct 30 02:43:13.163848 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.163773 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="563a9420-524b-48cc-9e77-2b6001604841" containerName="istio-proxy" Oct 30 02:43:13.163848 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.163810 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c5e8ed90-b3fb-4374-9380-733684ba2cb9" containerName="revoke-keys" Oct 30 02:43:13.163848 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.163816 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e8ed90-b3fb-4374-9380-733684ba2cb9" containerName="revoke-keys" Oct 30 02:43:13.163959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.163907 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e8ed90-b3fb-4374-9380-733684ba2cb9" containerName="revoke-keys" Oct 30 02:43:13.163959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.163918 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="563a9420-524b-48cc-9e77-2b6001604841" containerName="istio-proxy" Oct 30 02:43:13.167154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.167133 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx" Oct 30 02:43:13.179815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.179790 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:43:13.180120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.180072 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:43:13.190375 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.190343 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:43:13.190375 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.190376 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:43:13.194508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.194481 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx"] Oct 30 02:43:13.257156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.257119 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxd95\" (UniqueName: \"kubernetes.io/projected/87bd750b-f9a3-49b0-a1f5-fbbf12033834-kube-api-access-bxd95\") pod \"maas-api-revoke-sub-2dfa458b-k58xx\" (UID: \"87bd750b-f9a3-49b0-a1f5-fbbf12033834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx" Oct 30 02:43:13.257156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.257160 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bd750b-f9a3-49b0-a1f5-fbbf12033834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dfa458b-k58xx\" (UID: \"87bd750b-f9a3-49b0-a1f5-fbbf12033834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx" Oct 30 02:43:13.257397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.257232 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bd750b-f9a3-49b0-a1f5-fbbf12033834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dfa458b-k58xx\" (UID: \"87bd750b-f9a3-49b0-a1f5-fbbf12033834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx" Oct 30 02:43:13.358314 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.358266 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxd95\" (UniqueName: \"kubernetes.io/projected/87bd750b-f9a3-49b0-a1f5-fbbf12033834-kube-api-access-bxd95\") pod \"maas-api-revoke-sub-2dfa458b-k58xx\" (UID: \"87bd750b-f9a3-49b0-a1f5-fbbf12033834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx" Oct 30 02:43:13.358314 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.358317 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bd750b-f9a3-49b0-a1f5-fbbf12033834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dfa458b-k58xx\" (UID: \"87bd750b-f9a3-49b0-a1f5-fbbf12033834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx" Oct 30 02:43:13.358864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.358409 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bd750b-f9a3-49b0-a1f5-fbbf12033834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dfa458b-k58xx\" (UID: \"87bd750b-f9a3-49b0-a1f5-fbbf12033834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx" Oct 30 02:43:13.359479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.359238 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bd750b-f9a3-49b0-a1f5-fbbf12033834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dfa458b-k58xx\" (UID: \"87bd750b-f9a3-49b0-a1f5-fbbf12033834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx" Oct 30 02:43:13.360945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.360919 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bd750b-f9a3-49b0-a1f5-fbbf12033834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dfa458b-k58xx\" (UID: \"87bd750b-f9a3-49b0-a1f5-fbbf12033834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx" Oct 30 02:43:13.380432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.380399 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxd95\" (UniqueName: \"kubernetes.io/projected/87bd750b-f9a3-49b0-a1f5-fbbf12033834-kube-api-access-bxd95\") pod \"maas-api-revoke-sub-2dfa458b-k58xx\" (UID: \"87bd750b-f9a3-49b0-a1f5-fbbf12033834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx" Oct 30 02:43:13.481976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.481889 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx" Oct 30 02:43:13.768778 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.768699 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:13.793924 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.793890 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx"] Oct 30 02:43:13.798452 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:43:13.798420 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87bd750b_f9a3_49b0_a1f5_fbbf12033834.slice/crio-7f85405fc0ef7319a11bde43a64af7c8f954ce8aacac70be2144f32939192bb0 WatchSource:0}: Error finding container 7f85405fc0ef7319a11bde43a64af7c8f954ce8aacac70be2144f32939192bb0: Status 404 returned error can't find the container with id 7f85405fc0ef7319a11bde43a64af7c8f954ce8aacac70be2144f32939192bb0 Oct 30 02:43:13.927849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:13.927803 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx" event={"ID":"87bd750b-f9a3-49b0-a1f5-fbbf12033834","Type":"ContainerStarted","Data":"7f85405fc0ef7319a11bde43a64af7c8f954ce8aacac70be2144f32939192bb0"} Oct 30 02:43:14.940867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:14.940823 5998 generic.go:356] "Generic (PLEG): container finished" podID="87bd750b-f9a3-49b0-a1f5-fbbf12033834" containerID="eb08334a42960d3c89fa9644c522ca309957bade642974f535f21b5d73608fb2" exitCode=0 Oct 30 02:43:14.941392 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:14.940933 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx" event={"ID":"87bd750b-f9a3-49b0-a1f5-fbbf12033834","Type":"ContainerDied","Data":"eb08334a42960d3c89fa9644c522ca309957bade642974f535f21b5d73608fb2"} Oct 30 02:43:14.960130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:14.958154 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:15.075799 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:15.073651 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:15.489111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:15.481594 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:15.784486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:15.784398 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:16.115457 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:16.115428 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx" Oct 30 02:43:16.213246 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:16.213207 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87bd750b-f9a3-49b0-a1f5-fbbf12033834-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bd750b-f9a3-49b0-a1f5-fbbf12033834-maas-api-openshift-ca\") pod \"87bd750b-f9a3-49b0-a1f5-fbbf12033834\" (UID: \"87bd750b-f9a3-49b0-a1f5-fbbf12033834\") " Oct 30 02:43:16.213454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:16.213329 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87bd750b-f9a3-49b0-a1f5-fbbf12033834-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bd750b-f9a3-49b0-a1f5-fbbf12033834-maas-api-xks-ca\") pod \"87bd750b-f9a3-49b0-a1f5-fbbf12033834\" (UID: \"87bd750b-f9a3-49b0-a1f5-fbbf12033834\") " Oct 30 02:43:16.213454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:16.213444 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87bd750b-f9a3-49b0-a1f5-fbbf12033834-kube-api-access-bxd95\" (UniqueName: \"kubernetes.io/projected/87bd750b-f9a3-49b0-a1f5-fbbf12033834-kube-api-access-bxd95\") pod \"87bd750b-f9a3-49b0-a1f5-fbbf12033834\" (UID: \"87bd750b-f9a3-49b0-a1f5-fbbf12033834\") " Oct 30 02:43:16.214394 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:16.213703 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87bd750b-f9a3-49b0-a1f5-fbbf12033834-maas-api-openshift-ca" pod "87bd750b-f9a3-49b0-a1f5-fbbf12033834" (UID: "87bd750b-f9a3-49b0-a1f5-fbbf12033834"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:43:16.214394 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:16.214036 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87bd750b-f9a3-49b0-a1f5-fbbf12033834-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:16.216359 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:16.216326 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87bd750b-f9a3-49b0-a1f5-fbbf12033834-kube-api-access-bxd95" pod "87bd750b-f9a3-49b0-a1f5-fbbf12033834" (UID: "87bd750b-f9a3-49b0-a1f5-fbbf12033834"). InnerVolumeSpecName "kube-api-access-bxd95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:43:16.216663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:16.216626 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87bd750b-f9a3-49b0-a1f5-fbbf12033834-maas-api-xks-ca" pod "87bd750b-f9a3-49b0-a1f5-fbbf12033834" (UID: "87bd750b-f9a3-49b0-a1f5-fbbf12033834"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:43:16.315372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:16.315326 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87bd750b-f9a3-49b0-a1f5-fbbf12033834-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:16.315372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:16.315363 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxd95\" (UniqueName: \"kubernetes.io/projected/87bd750b-f9a3-49b0-a1f5-fbbf12033834-kube-api-access-bxd95\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:16.958776 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:16.958741 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx" Oct 30 02:43:16.958776 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:16.958765 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx" event={"ID":"87bd750b-f9a3-49b0-a1f5-fbbf12033834","Type":"ContainerDied","Data":"7f85405fc0ef7319a11bde43a64af7c8f954ce8aacac70be2144f32939192bb0"} Oct 30 02:43:16.959080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:16.958817 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f85405fc0ef7319a11bde43a64af7c8f954ce8aacac70be2144f32939192bb0" Oct 30 02:43:17.233578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:17.231912 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:18.995871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:18.995837 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:20.673970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:20.673933 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:22.148159 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.148121 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" containerID="cri-o://f49533f7d96f11b6b6bba1a67e941e25792d99ae7e53fa4f8301932ab34249ca" gracePeriod=45 Oct 30 02:43:22.159578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.159539 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" containerID="cri-o://a2b68025459301444789f6d51fa02f0936be8351d5f087b43a269cbbf8473a27" gracePeriod=45 Oct 30 02:43:22.172659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.172626 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" containerID="cri-o://75d9abe139b45165f1b87576b669136870cbeb1931bd3c2ee65464aec131419a" gracePeriod=45 Oct 30 02:43:22.383223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.383076 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:22.486513 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.486456 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 02:43:22.506676 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.506645 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 02:43:22.521275 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.521242 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 02:43:22.613110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.613058 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-tmp-dir\") pod \"161dcfca-cef5-45b8-983d-3bffa46cc52d\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " Oct 30 02:43:22.613312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.613187 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-dshm\") pod \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " Oct 30 02:43:22.613312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.613284 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aafa0b2-1b3f-479a-89a5-2041cf887a21-kube-api-access-p2lvp\" (UniqueName: \"kubernetes.io/projected/3aafa0b2-1b3f-479a-89a5-2041cf887a21-kube-api-access-p2lvp\") pod \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " Oct 30 02:43:22.613446 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.613349 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-dshm\") pod \"161dcfca-cef5-45b8-983d-3bffa46cc52d\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " Oct 30 02:43:22.613446 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.613353 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-tmp-dir" pod "161dcfca-cef5-45b8-983d-3bffa46cc52d" (UID: "161dcfca-cef5-45b8-983d-3bffa46cc52d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:22.613446 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.613441 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-home\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-home\") pod \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " Oct 30 02:43:22.613620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.613470 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aafa0b2-1b3f-479a-89a5-2041cf887a21-tls-certs\" (UniqueName: \"kubernetes.io/secret/3aafa0b2-1b3f-479a-89a5-2041cf887a21-tls-certs\") pod \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " Oct 30 02:43:22.613620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.613505 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-model-cache\") pod \"161dcfca-cef5-45b8-983d-3bffa46cc52d\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " Oct 30 02:43:22.613620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.613549 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-tmp-dir\") pod \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " Oct 30 02:43:22.613620 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.613603 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/161dcfca-cef5-45b8-983d-3bffa46cc52d-tls-certs\" (UniqueName: \"kubernetes.io/secret/161dcfca-cef5-45b8-983d-3bffa46cc52d-tls-certs\") pod \"161dcfca-cef5-45b8-983d-3bffa46cc52d\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " Oct 30 02:43:22.613831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.613640 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/161dcfca-cef5-45b8-983d-3bffa46cc52d-kube-api-access-c6h2r\" (UniqueName: \"kubernetes.io/projected/161dcfca-cef5-45b8-983d-3bffa46cc52d-kube-api-access-c6h2r\") pod \"161dcfca-cef5-45b8-983d-3bffa46cc52d\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " Oct 30 02:43:22.613831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.613720 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-model-cache\") pod \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\" (UID: \"3aafa0b2-1b3f-479a-89a5-2041cf887a21\") " Oct 30 02:43:22.613831 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.613786 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-home\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-home\") pod \"161dcfca-cef5-45b8-983d-3bffa46cc52d\" (UID: \"161dcfca-cef5-45b8-983d-3bffa46cc52d\") " Oct 30 02:43:22.615665 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.614317 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.615665 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.614532 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-home" pod "161dcfca-cef5-45b8-983d-3bffa46cc52d" (UID: "161dcfca-cef5-45b8-983d-3bffa46cc52d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:22.615665 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.615108 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-model-cache" pod "3aafa0b2-1b3f-479a-89a5-2041cf887a21" (UID: "3aafa0b2-1b3f-479a-89a5-2041cf887a21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:22.615665 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.615363 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-model-cache" pod "161dcfca-cef5-45b8-983d-3bffa46cc52d" (UID: "161dcfca-cef5-45b8-983d-3bffa46cc52d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:22.615665 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.615565 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-tmp-dir" pod "3aafa0b2-1b3f-479a-89a5-2041cf887a21" (UID: "3aafa0b2-1b3f-479a-89a5-2041cf887a21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:22.615665 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.615638 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-home" pod "3aafa0b2-1b3f-479a-89a5-2041cf887a21" (UID: "3aafa0b2-1b3f-479a-89a5-2041cf887a21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:22.616752 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.616525 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aafa0b2-1b3f-479a-89a5-2041cf887a21-tls-certs" pod "3aafa0b2-1b3f-479a-89a5-2041cf887a21" (UID: "3aafa0b2-1b3f-479a-89a5-2041cf887a21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:43:22.616752 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.616722 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-dshm" pod "3aafa0b2-1b3f-479a-89a5-2041cf887a21" (UID: "3aafa0b2-1b3f-479a-89a5-2041cf887a21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:22.617409 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.617359 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-dshm" pod "161dcfca-cef5-45b8-983d-3bffa46cc52d" (UID: "161dcfca-cef5-45b8-983d-3bffa46cc52d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:22.618713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.618687 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/161dcfca-cef5-45b8-983d-3bffa46cc52d-tls-certs" pod "161dcfca-cef5-45b8-983d-3bffa46cc52d" (UID: "161dcfca-cef5-45b8-983d-3bffa46cc52d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:43:22.618801 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.618727 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161dcfca-cef5-45b8-983d-3bffa46cc52d-kube-api-access-c6h2r" pod "161dcfca-cef5-45b8-983d-3bffa46cc52d" (UID: "161dcfca-cef5-45b8-983d-3bffa46cc52d"). InnerVolumeSpecName "kube-api-access-c6h2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:43:22.619402 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.619376 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aafa0b2-1b3f-479a-89a5-2041cf887a21-kube-api-access-p2lvp" pod "3aafa0b2-1b3f-479a-89a5-2041cf887a21" (UID: "3aafa0b2-1b3f-479a-89a5-2041cf887a21"). InnerVolumeSpecName "kube-api-access-p2lvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:43:22.715714 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.715562 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-tls-certs\") pod \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " Oct 30 02:43:22.716134 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.715972 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-model-cache\") pod \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " Oct 30 02:43:22.716449 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.716278 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-dshm\") pod \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " Oct 30 02:43:22.716602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.716573 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-tmp-dir\") pod \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " Oct 30 02:43:22.716761 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.716648 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-home\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-home\") pod \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " Oct 30 02:43:22.716761 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.716722 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-model-cache" pod "448d43d9-9a90-4567-a9af-d5b9dd4c20ac" (UID: "448d43d9-9a90-4567-a9af-d5b9dd4c20ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:22.716761 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.716740 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-kube-api-access-mfd9t\" (UniqueName: \"kubernetes.io/projected/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-kube-api-access-mfd9t\") pod \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\" (UID: \"448d43d9-9a90-4567-a9af-d5b9dd4c20ac\") " Oct 30 02:43:22.717006 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.716958 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-tmp-dir" pod "448d43d9-9a90-4567-a9af-d5b9dd4c20ac" (UID: "448d43d9-9a90-4567-a9af-d5b9dd4c20ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:22.717255 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.717233 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-home" pod "448d43d9-9a90-4567-a9af-d5b9dd4c20ac" (UID: "448d43d9-9a90-4567-a9af-d5b9dd4c20ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:22.717551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.717521 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.717551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.717552 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.717706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.717566 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.717706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.717582 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.717706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.717598 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2lvp\" (UniqueName: \"kubernetes.io/projected/3aafa0b2-1b3f-479a-89a5-2041cf887a21-kube-api-access-p2lvp\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.717706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.717613 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.717706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.717627 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.717706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.717640 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3aafa0b2-1b3f-479a-89a5-2041cf887a21-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.717706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.717654 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/161dcfca-cef5-45b8-983d-3bffa46cc52d-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.717706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.717667 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.717706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.717681 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/161dcfca-cef5-45b8-983d-3bffa46cc52d-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.717706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.717696 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6h2r\" (UniqueName: \"kubernetes.io/projected/161dcfca-cef5-45b8-983d-3bffa46cc52d-kube-api-access-c6h2r\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.717706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.717708 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.718061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.717722 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3aafa0b2-1b3f-479a-89a5-2041cf887a21-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.718828 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.718808 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-dshm" pod "448d43d9-9a90-4567-a9af-d5b9dd4c20ac" (UID: "448d43d9-9a90-4567-a9af-d5b9dd4c20ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:43:22.719008 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.718983 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-tls-certs" pod "448d43d9-9a90-4567-a9af-d5b9dd4c20ac" (UID: "448d43d9-9a90-4567-a9af-d5b9dd4c20ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:43:22.719159 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.719129 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-kube-api-access-mfd9t" pod "448d43d9-9a90-4567-a9af-d5b9dd4c20ac" (UID: "448d43d9-9a90-4567-a9af-d5b9dd4c20ac"). InnerVolumeSpecName "kube-api-access-mfd9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:43:22.819419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.819379 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfd9t\" (UniqueName: \"kubernetes.io/projected/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-kube-api-access-mfd9t\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.819419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.819412 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:22.819419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:22.819424 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/448d43d9-9a90-4567-a9af-d5b9dd4c20ac-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:43:23.009609 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.009512 5998 generic.go:356] "Generic (PLEG): container finished" podID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerID="f49533f7d96f11b6b6bba1a67e941e25792d99ae7e53fa4f8301932ab34249ca" exitCode=0 Oct 30 02:43:23.009609 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.009556 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" Oct 30 02:43:23.009609 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.009593 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" event={"ID":"3aafa0b2-1b3f-479a-89a5-2041cf887a21","Type":"ContainerDied","Data":"f49533f7d96f11b6b6bba1a67e941e25792d99ae7e53fa4f8301932ab34249ca"} Oct 30 02:43:23.009891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.009642 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d" event={"ID":"3aafa0b2-1b3f-479a-89a5-2041cf887a21","Type":"ContainerDied","Data":"0867b759325d046a77adaed49bc94ca1a5937502d1a4ace5fc98f87408f8fae4"} Oct 30 02:43:23.009891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.009665 5998 scope.go:122] "RemoveContainer" containerID="f49533f7d96f11b6b6bba1a67e941e25792d99ae7e53fa4f8301932ab34249ca" Oct 30 02:43:23.011464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.011438 5998 generic.go:356] "Generic (PLEG): container finished" podID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerID="75d9abe139b45165f1b87576b669136870cbeb1931bd3c2ee65464aec131419a" exitCode=0 Oct 30 02:43:23.011576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.011498 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" Oct 30 02:43:23.011576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.011526 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" event={"ID":"448d43d9-9a90-4567-a9af-d5b9dd4c20ac","Type":"ContainerDied","Data":"75d9abe139b45165f1b87576b669136870cbeb1931bd3c2ee65464aec131419a"} Oct 30 02:43:23.011576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.011567 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv" event={"ID":"448d43d9-9a90-4567-a9af-d5b9dd4c20ac","Type":"ContainerDied","Data":"46419f88e3022f204dba9afb0cc8710e2458015b28265368f7a10d1d0df4c9fa"} Oct 30 02:43:23.013222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.013199 5998 generic.go:356] "Generic (PLEG): container finished" podID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerID="a2b68025459301444789f6d51fa02f0936be8351d5f087b43a269cbbf8473a27" exitCode=0 Oct 30 02:43:23.013336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.013264 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" event={"ID":"161dcfca-cef5-45b8-983d-3bffa46cc52d","Type":"ContainerDied","Data":"a2b68025459301444789f6d51fa02f0936be8351d5f087b43a269cbbf8473a27"} Oct 30 02:43:23.013336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.013293 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" event={"ID":"161dcfca-cef5-45b8-983d-3bffa46cc52d","Type":"ContainerDied","Data":"9031be1ac4d0f3e881bca2bdfa8607875c2f4a7a8de29cae6e58b890d145303c"} Oct 30 02:43:23.013466 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.013350 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7" Oct 30 02:43:23.024229 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.024198 5998 scope.go:122] "RemoveContainer" containerID="f49533f7d96f11b6b6bba1a67e941e25792d99ae7e53fa4f8301932ab34249ca" Oct 30 02:43:23.024543 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:43:23.024520 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f49533f7d96f11b6b6bba1a67e941e25792d99ae7e53fa4f8301932ab34249ca\": container with ID starting with f49533f7d96f11b6b6bba1a67e941e25792d99ae7e53fa4f8301932ab34249ca not found: ID does not exist" containerID="f49533f7d96f11b6b6bba1a67e941e25792d99ae7e53fa4f8301932ab34249ca" Oct 30 02:43:23.024627 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.024556 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f49533f7d96f11b6b6bba1a67e941e25792d99ae7e53fa4f8301932ab34249ca"} err="failed to get container status \"f49533f7d96f11b6b6bba1a67e941e25792d99ae7e53fa4f8301932ab34249ca\": rpc error: code = NotFound desc = could not find container \"f49533f7d96f11b6b6bba1a67e941e25792d99ae7e53fa4f8301932ab34249ca\": container with ID starting with f49533f7d96f11b6b6bba1a67e941e25792d99ae7e53fa4f8301932ab34249ca not found: ID does not exist" Oct 30 02:43:23.024627 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.024580 5998 scope.go:122] "RemoveContainer" containerID="75d9abe139b45165f1b87576b669136870cbeb1931bd3c2ee65464aec131419a" Oct 30 02:43:23.034666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.034645 5998 scope.go:122] "RemoveContainer" containerID="75d9abe139b45165f1b87576b669136870cbeb1931bd3c2ee65464aec131419a" Oct 30 02:43:23.035009 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:43:23.034989 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d9abe139b45165f1b87576b669136870cbeb1931bd3c2ee65464aec131419a\": container with ID starting with 75d9abe139b45165f1b87576b669136870cbeb1931bd3c2ee65464aec131419a not found: ID does not exist" containerID="75d9abe139b45165f1b87576b669136870cbeb1931bd3c2ee65464aec131419a" Oct 30 02:43:23.035064 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.035019 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d9abe139b45165f1b87576b669136870cbeb1931bd3c2ee65464aec131419a"} err="failed to get container status \"75d9abe139b45165f1b87576b669136870cbeb1931bd3c2ee65464aec131419a\": rpc error: code = NotFound desc = could not find container \"75d9abe139b45165f1b87576b669136870cbeb1931bd3c2ee65464aec131419a\": container with ID starting with 75d9abe139b45165f1b87576b669136870cbeb1931bd3c2ee65464aec131419a not found: ID does not exist" Oct 30 02:43:23.035124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.035067 5998 scope.go:122] "RemoveContainer" containerID="a2b68025459301444789f6d51fa02f0936be8351d5f087b43a269cbbf8473a27" Oct 30 02:43:23.044860 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.044836 5998 scope.go:122] "RemoveContainer" containerID="a2b68025459301444789f6d51fa02f0936be8351d5f087b43a269cbbf8473a27" Oct 30 02:43:23.045198 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:43:23.045176 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b68025459301444789f6d51fa02f0936be8351d5f087b43a269cbbf8473a27\": container with ID starting with a2b68025459301444789f6d51fa02f0936be8351d5f087b43a269cbbf8473a27 not found: ID does not exist" containerID="a2b68025459301444789f6d51fa02f0936be8351d5f087b43a269cbbf8473a27" Oct 30 02:43:23.045269 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.045205 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b68025459301444789f6d51fa02f0936be8351d5f087b43a269cbbf8473a27"} err="failed to get container status \"a2b68025459301444789f6d51fa02f0936be8351d5f087b43a269cbbf8473a27\": rpc error: code = NotFound desc = could not find container \"a2b68025459301444789f6d51fa02f0936be8351d5f087b43a269cbbf8473a27\": container with ID starting with a2b68025459301444789f6d51fa02f0936be8351d5f087b43a269cbbf8473a27 not found: ID does not exist" Oct 30 02:43:23.071328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.071288 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d"] Oct 30 02:43:23.081135 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.081105 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/distinct-w1-f6f4f0-kserve-5dfd8cb984-ldf5d"] Oct 30 02:43:23.121425 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.121389 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv"] Oct 30 02:43:23.127778 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.126353 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/simulator-w1-f6f4f0-kserve-7879b5f6fb-qvlnv"] Oct 30 02:43:23.157813 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.157775 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7"] Oct 30 02:43:23.163116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.163067 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-f6f4f0/premium-w1-f6f4f0-kserve-674f58d54f-gqpq7"] Oct 30 02:43:23.265429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:23.265345 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:24.070584 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:24.070467 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:24.538827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:24.538789 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" path="/var/lib/kubelet/pods/161dcfca-cef5-45b8-983d-3bffa46cc52d/volumes" Oct 30 02:43:24.539389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:24.539260 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" path="/var/lib/kubelet/pods/3aafa0b2-1b3f-479a-89a5-2041cf887a21/volumes" Oct 30 02:43:24.539676 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:24.539659 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" path="/var/lib/kubelet/pods/448d43d9-9a90-4567-a9af-d5b9dd4c20ac/volumes" Oct 30 02:43:25.217634 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:43:25.217596 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:43:25.661558 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:25.661521 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:26.711287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:26.711247 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:27.151053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:27.150967 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:28.829902 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:28.829864 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:30.140154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:30.140108 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:30.348423 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:30.348388 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:31.904154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:31.904113 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:33.558686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:33.558641 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:34.442309 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:34.442272 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:43:35.078578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:35.078477 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:36.810671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:36.810243 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:38.263309 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:38.263272 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:39.837544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:39.837500 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:41.252947 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:41.252912 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:41.513610 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:41.513509 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:42.986704 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:42.986672 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:44.598167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:44.598122 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:45.255082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:45.254998 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:45.375310 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:45.375273 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:45.767184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:45.767145 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:46.295221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:46.295177 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:47.715493 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:47.715455 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:48.468420 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:48.468382 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:50.179806 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:50.179771 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:51.774636 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:51.774578 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:52.726125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:52.726070 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:52.832142 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:52.832103 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:53.335975 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:53.335935 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:54.544978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:54.544937 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:54.845580 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:54.845497 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:55.221020 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:43:55.220979 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:43:56.409577 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:56.409544 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:57.452378 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:57.452339 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:57.567434 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:57.567400 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:58.047259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:58.047224 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:58.276354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:58.276319 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:58.653158 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:58.653121 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx"] Oct 30 02:43:58.653655 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:58.653380 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" podUID="e34d09e7-ec9d-4160-9c74-13b2ccf8d871" containerName="istio-proxy" containerID="cri-o://b8c2f3f0f63a3531d1e6b740ccfc756a7c82191b4d723e0a46e3fc30f8d37830" gracePeriod=30 Oct 30 02:43:58.667632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:58.667590 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:43:59.390793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:59.390755 5998 patch_prober.go:29] interesting pod/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:43:59.390981 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:59.390833 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" podUID="e34d09e7-ec9d-4160-9c74-13b2ccf8d871" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:43:59.949669 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:43:59.949629 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:01.526719 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:01.526682 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:03.077014 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:03.076972 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:03.949692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:03.949662 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 02:44:04.082703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.082666 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-envoy\") pod \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " Oct 30 02:44:04.083203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.082725 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-podinfo\") pod \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " Oct 30 02:44:04.083203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.082758 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-data\") pod \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " Oct 30 02:44:04.083203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.082782 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-workload-socket\") pod \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " Oct 30 02:44:04.083203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.082836 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-kube-api-access-sxmqr\" (UniqueName: \"kubernetes.io/projected/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-kube-api-access-sxmqr\") pod \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " Oct 30 02:44:04.083203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.082947 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-credential-socket\") pod \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " Oct 30 02:44:04.083203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.082996 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-token\" (UniqueName: \"kubernetes.io/projected/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-token\") pod \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " Oct 30 02:44:04.083203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.083067 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istiod-ca-cert\") pod \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " Oct 30 02:44:04.083203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.083164 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-data" pod "e34d09e7-ec9d-4160-9c74-13b2ccf8d871" (UID: "e34d09e7-ec9d-4160-9c74-13b2ccf8d871"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:44:04.083203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.083199 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-workload-certs\") pod \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\" (UID: \"e34d09e7-ec9d-4160-9c74-13b2ccf8d871\") " Oct 30 02:44:04.083744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.083175 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-workload-socket" pod "e34d09e7-ec9d-4160-9c74-13b2ccf8d871" (UID: "e34d09e7-ec9d-4160-9c74-13b2ccf8d871"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:44:04.083744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.083288 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-credential-socket" pod "e34d09e7-ec9d-4160-9c74-13b2ccf8d871" (UID: "e34d09e7-ec9d-4160-9c74-13b2ccf8d871"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:44:04.083744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.083518 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istiod-ca-cert" pod "e34d09e7-ec9d-4160-9c74-13b2ccf8d871" (UID: "e34d09e7-ec9d-4160-9c74-13b2ccf8d871"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:44:04.083744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.083555 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-workload-certs" pod "e34d09e7-ec9d-4160-9c74-13b2ccf8d871" (UID: "e34d09e7-ec9d-4160-9c74-13b2ccf8d871"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:44:04.083984 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.083839 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:44:04.083984 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.083862 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:44:04.083984 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.083877 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:44:04.083984 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.083890 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:44:04.083984 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.083903 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:44:04.085186 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.085153 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-envoy" pod "e34d09e7-ec9d-4160-9c74-13b2ccf8d871" (UID: "e34d09e7-ec9d-4160-9c74-13b2ccf8d871"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:44:04.085315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.085267 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-podinfo" pod "e34d09e7-ec9d-4160-9c74-13b2ccf8d871" (UID: "e34d09e7-ec9d-4160-9c74-13b2ccf8d871"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:44:04.085315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.085293 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-kube-api-access-sxmqr" pod "e34d09e7-ec9d-4160-9c74-13b2ccf8d871" (UID: "e34d09e7-ec9d-4160-9c74-13b2ccf8d871"). InnerVolumeSpecName "kube-api-access-sxmqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:44:04.085441 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.085311 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-token" pod "e34d09e7-ec9d-4160-9c74-13b2ccf8d871" (UID: "e34d09e7-ec9d-4160-9c74-13b2ccf8d871"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:44:04.185069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.184980 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:44:04.185069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.185015 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:44:04.185069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.185024 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:44:04.185069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.185035 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxmqr\" (UniqueName: \"kubernetes.io/projected/e34d09e7-ec9d-4160-9c74-13b2ccf8d871-kube-api-access-sxmqr\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:44:04.316916 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.316875 5998 generic.go:356] "Generic (PLEG): container finished" podID="e34d09e7-ec9d-4160-9c74-13b2ccf8d871" containerID="b8c2f3f0f63a3531d1e6b740ccfc756a7c82191b4d723e0a46e3fc30f8d37830" exitCode=0 Oct 30 02:44:04.317173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.316927 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" Oct 30 02:44:04.317173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.316966 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" event={"ID":"e34d09e7-ec9d-4160-9c74-13b2ccf8d871","Type":"ContainerDied","Data":"b8c2f3f0f63a3531d1e6b740ccfc756a7c82191b4d723e0a46e3fc30f8d37830"} Oct 30 02:44:04.317173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.317038 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx" event={"ID":"e34d09e7-ec9d-4160-9c74-13b2ccf8d871","Type":"ContainerDied","Data":"2482b413756599445b0c5c358b5cd336c6f1cfc9f32718d500435702cb5ee18a"} Oct 30 02:44:04.317173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.317074 5998 scope.go:122] "RemoveContainer" containerID="b8c2f3f0f63a3531d1e6b740ccfc756a7c82191b4d723e0a46e3fc30f8d37830" Oct 30 02:44:04.330755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.330732 5998 scope.go:122] "RemoveContainer" containerID="b8c2f3f0f63a3531d1e6b740ccfc756a7c82191b4d723e0a46e3fc30f8d37830" Oct 30 02:44:04.331118 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:44:04.331081 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c2f3f0f63a3531d1e6b740ccfc756a7c82191b4d723e0a46e3fc30f8d37830\": container with ID starting with b8c2f3f0f63a3531d1e6b740ccfc756a7c82191b4d723e0a46e3fc30f8d37830 not found: ID does not exist" containerID="b8c2f3f0f63a3531d1e6b740ccfc756a7c82191b4d723e0a46e3fc30f8d37830" Oct 30 02:44:04.331194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.331133 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c2f3f0f63a3531d1e6b740ccfc756a7c82191b4d723e0a46e3fc30f8d37830"} err="failed to get container status \"b8c2f3f0f63a3531d1e6b740ccfc756a7c82191b4d723e0a46e3fc30f8d37830\": rpc error: code = NotFound desc = could not find container \"b8c2f3f0f63a3531d1e6b740ccfc756a7c82191b4d723e0a46e3fc30f8d37830\": container with ID starting with b8c2f3f0f63a3531d1e6b740ccfc756a7c82191b4d723e0a46e3fc30f8d37830 not found: ID does not exist" Oct 30 02:44:04.364202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.364171 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx"] Oct 30 02:44:04.374583 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.374552 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-f6f4f0-openshift-default-78469759d7-7cvxx"] Oct 30 02:44:04.540872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.540784 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e34d09e7-ec9d-4160-9c74-13b2ccf8d871" path="/var/lib/kubelet/pods/e34d09e7-ec9d-4160-9c74-13b2ccf8d871/volumes" Oct 30 02:44:04.592267 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:04.592234 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:05.677686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:05.677654 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:06.171900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:06.171870 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:07.664619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:07.664584 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:09.172744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:09.172706 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:10.693876 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:10.693838 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:12.282511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:12.282477 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:13.762075 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:13.762033 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:15.250878 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:15.250840 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:16.783232 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:16.783193 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:18.397631 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:18.397590 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:19.966165 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:19.966127 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:21.464447 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:21.464368 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:23.119044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:23.119009 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:23.796883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:23.796843 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:44:24.578456 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:24.578419 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:25.223246 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:44:25.223201 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:44:26.145448 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:26.145396 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:27.754601 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:27.754557 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:29.309612 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:29.309536 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:30.874863 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:30.874809 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:32.436405 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:32.436365 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:34.141941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:34.141897 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:34.858735 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:34.858697 5998 scope.go:122] "RemoveContainer" containerID="d3b641a80ce11c270a6c07b6217c02580315a1ae126b1d23b59e054e3a962d81" Oct 30 02:44:34.869023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:34.868994 5998 scope.go:122] "RemoveContainer" containerID="d52f8bfbdb6fc2296bafc9735a43b71b2e5147294c78ccf65a5f412eb4396df2" Oct 30 02:44:34.878100 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:34.878062 5998 scope.go:122] "RemoveContainer" containerID="c953aded0e0ce0adac69aee0a8f1184b5dd6e8705fe03b6cc1a445852ea448a3" Oct 30 02:44:35.788184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:35.788145 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:37.368649 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:37.368589 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:38.993958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:38.993910 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:40.042020 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:40.041975 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq"] Oct 30 02:44:40.052061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:40.052023 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b273dc16-8h7xq"] Oct 30 02:44:40.540187 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:40.540144 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de41b56f-ff6f-4e31-af10-acf732942839" path="/var/lib/kubelet/pods/de41b56f-ff6f-4e31-af10-acf732942839/volumes" Oct 30 02:44:40.686877 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:40.686842 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:41.370326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:41.370268 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:42.165460 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:42.165333 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:43.674683 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:43.674644 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:45.179659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:45.179621 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:46.657037 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:46.657000 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:47.624282 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:47.624241 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:44:48.380544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:48.380476 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:50.041335 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:50.041294 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:51.553889 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:51.553856 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:53.152541 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:53.152506 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:54.025228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.025188 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp"] Oct 30 02:44:54.025985 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.025958 5998 cpu_manager.go:414] "containerMap: removing container" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" Oct 30 02:44:54.025985 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.025984 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" Oct 30 02:44:54.026222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.026002 5998 cpu_manager.go:414] "containerMap: removing container" podUID="563a9420-524b-48cc-9e77-2b6001604841" containerName="istio-proxy" Oct 30 02:44:54.026222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.026012 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="563a9420-524b-48cc-9e77-2b6001604841" containerName="istio-proxy" Oct 30 02:44:54.026222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.026029 5998 cpu_manager.go:414] "containerMap: removing container" podUID="e34d09e7-ec9d-4160-9c74-13b2ccf8d871" containerName="istio-proxy" Oct 30 02:44:54.026222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.026037 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e34d09e7-ec9d-4160-9c74-13b2ccf8d871" containerName="istio-proxy" Oct 30 02:44:54.026222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.026070 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" Oct 30 02:44:54.026222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.026079 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" Oct 30 02:44:54.026222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.026108 5998 cpu_manager.go:414] "containerMap: removing container" podUID="87bd750b-f9a3-49b0-a1f5-fbbf12033834" containerName="revoke-keys" Oct 30 02:44:54.026222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.026116 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87bd750b-f9a3-49b0-a1f5-fbbf12033834" containerName="revoke-keys" Oct 30 02:44:54.026222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.026145 5998 cpu_manager.go:414] "containerMap: removing container" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" Oct 30 02:44:54.026222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.026152 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" Oct 30 02:44:54.026821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.026288 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="448d43d9-9a90-4567-a9af-d5b9dd4c20ac" containerName="main" Oct 30 02:44:54.026821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.026301 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="563a9420-524b-48cc-9e77-2b6001604841" containerName="istio-proxy" Oct 30 02:44:54.026821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.026311 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="e34d09e7-ec9d-4160-9c74-13b2ccf8d871" containerName="istio-proxy" Oct 30 02:44:54.026821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.026325 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aafa0b2-1b3f-479a-89a5-2041cf887a21" containerName="main" Oct 30 02:44:54.026821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.026335 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="87bd750b-f9a3-49b0-a1f5-fbbf12033834" containerName="revoke-keys" Oct 30 02:44:54.026821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.026348 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="161dcfca-cef5-45b8-983d-3bffa46cc52d" containerName="main" Oct 30 02:44:54.030483 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.030459 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp" Oct 30 02:44:54.050860 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.050825 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp"] Oct 30 02:44:54.055711 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.055683 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:44:54.055872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.055704 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:44:54.055872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.055746 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:44:54.055872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.055853 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:44:54.159935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.159896 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f42c813-e505-4819-b8b5-f080cf92ffd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a99cb651-4gtdp\" (UID: \"4f42c813-e505-4819-b8b5-f080cf92ffd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp" Oct 30 02:44:54.160366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.160080 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw7nz\" (UniqueName: \"kubernetes.io/projected/4f42c813-e505-4819-b8b5-f080cf92ffd6-kube-api-access-kw7nz\") pod \"maas-api-revoke-sub-a99cb651-4gtdp\" (UID: \"4f42c813-e505-4819-b8b5-f080cf92ffd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp" Oct 30 02:44:54.160366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.160213 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f42c813-e505-4819-b8b5-f080cf92ffd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a99cb651-4gtdp\" (UID: \"4f42c813-e505-4819-b8b5-f080cf92ffd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp" Oct 30 02:44:54.261172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.261125 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f42c813-e505-4819-b8b5-f080cf92ffd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a99cb651-4gtdp\" (UID: \"4f42c813-e505-4819-b8b5-f080cf92ffd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp" Oct 30 02:44:54.261370 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.261273 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw7nz\" (UniqueName: \"kubernetes.io/projected/4f42c813-e505-4819-b8b5-f080cf92ffd6-kube-api-access-kw7nz\") pod \"maas-api-revoke-sub-a99cb651-4gtdp\" (UID: \"4f42c813-e505-4819-b8b5-f080cf92ffd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp" Oct 30 02:44:54.261370 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.261338 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f42c813-e505-4819-b8b5-f080cf92ffd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a99cb651-4gtdp\" (UID: \"4f42c813-e505-4819-b8b5-f080cf92ffd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp" Oct 30 02:44:54.262029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.262005 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f42c813-e505-4819-b8b5-f080cf92ffd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a99cb651-4gtdp\" (UID: \"4f42c813-e505-4819-b8b5-f080cf92ffd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp" Oct 30 02:44:54.263670 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.263645 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f42c813-e505-4819-b8b5-f080cf92ffd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a99cb651-4gtdp\" (UID: \"4f42c813-e505-4819-b8b5-f080cf92ffd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp" Oct 30 02:44:54.283110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.283026 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw7nz\" (UniqueName: \"kubernetes.io/projected/4f42c813-e505-4819-b8b5-f080cf92ffd6-kube-api-access-kw7nz\") pod \"maas-api-revoke-sub-a99cb651-4gtdp\" (UID: \"4f42c813-e505-4819-b8b5-f080cf92ffd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp" Oct 30 02:44:54.346323 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.346271 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp" Oct 30 02:44:54.660283 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.660256 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp"] Oct 30 02:44:54.662710 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:44:54.662679 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f42c813_e505_4819_b8b5_f080cf92ffd6.slice/crio-8ce4fb72b3ae56fdcbd4248d3e4ac000ed2b38bcc54bbeafbda74bdd6af5c1de WatchSource:0}: Error finding container 8ce4fb72b3ae56fdcbd4248d3e4ac000ed2b38bcc54bbeafbda74bdd6af5c1de: Status 404 returned error can't find the container with id 8ce4fb72b3ae56fdcbd4248d3e4ac000ed2b38bcc54bbeafbda74bdd6af5c1de Oct 30 02:44:54.733278 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.733237 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp" event={"ID":"4f42c813-e505-4819-b8b5-f080cf92ffd6","Type":"ContainerStarted","Data":"8ce4fb72b3ae56fdcbd4248d3e4ac000ed2b38bcc54bbeafbda74bdd6af5c1de"} Oct 30 02:44:54.768958 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:54.768924 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:55.104415 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:55.104373 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x"] Oct 30 02:44:55.108386 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:55.108359 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x" Oct 30 02:44:55.128724 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:55.128694 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x"] Oct 30 02:44:55.219972 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:44:55.219882 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:44:55.274602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:55.274565 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3508546-4215-44b4-9f02-4c90461c1083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3625c7d7-fpd8x\" (UID: \"c3508546-4215-44b4-9f02-4c90461c1083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x" Oct 30 02:44:55.274830 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:55.274802 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3508546-4215-44b4-9f02-4c90461c1083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3625c7d7-fpd8x\" (UID: \"c3508546-4215-44b4-9f02-4c90461c1083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x" Oct 30 02:44:55.274915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:55.274900 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h24wq\" (UniqueName: \"kubernetes.io/projected/c3508546-4215-44b4-9f02-4c90461c1083-kube-api-access-h24wq\") pod \"maas-api-revoke-sub-3625c7d7-fpd8x\" (UID: \"c3508546-4215-44b4-9f02-4c90461c1083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x" Oct 30 02:44:55.375664 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:55.375623 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3508546-4215-44b4-9f02-4c90461c1083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3625c7d7-fpd8x\" (UID: \"c3508546-4215-44b4-9f02-4c90461c1083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x" Oct 30 02:44:55.375892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:55.375688 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h24wq\" (UniqueName: \"kubernetes.io/projected/c3508546-4215-44b4-9f02-4c90461c1083-kube-api-access-h24wq\") pod \"maas-api-revoke-sub-3625c7d7-fpd8x\" (UID: \"c3508546-4215-44b4-9f02-4c90461c1083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x" Oct 30 02:44:55.375892 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:55.375790 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3508546-4215-44b4-9f02-4c90461c1083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3625c7d7-fpd8x\" (UID: \"c3508546-4215-44b4-9f02-4c90461c1083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x" Oct 30 02:44:55.376514 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:55.376484 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3508546-4215-44b4-9f02-4c90461c1083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3625c7d7-fpd8x\" (UID: \"c3508546-4215-44b4-9f02-4c90461c1083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x" Oct 30 02:44:55.378479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:55.378457 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3508546-4215-44b4-9f02-4c90461c1083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3625c7d7-fpd8x\" (UID: \"c3508546-4215-44b4-9f02-4c90461c1083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x" Oct 30 02:44:55.397917 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:55.397882 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h24wq\" (UniqueName: \"kubernetes.io/projected/c3508546-4215-44b4-9f02-4c90461c1083-kube-api-access-h24wq\") pod \"maas-api-revoke-sub-3625c7d7-fpd8x\" (UID: \"c3508546-4215-44b4-9f02-4c90461c1083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x" Oct 30 02:44:55.426324 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:55.426289 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x" Oct 30 02:44:55.730845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:55.730809 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x"] Oct 30 02:44:55.733426 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:44:55.733388 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3508546_4215_44b4_9f02_4c90461c1083.slice/crio-76505d04751a8ea3ed3c9a6a6173338dd60306a209ff2bfdb1f96b2bd24cf9b4 WatchSource:0}: Error finding container 76505d04751a8ea3ed3c9a6a6173338dd60306a209ff2bfdb1f96b2bd24cf9b4: Status 404 returned error can't find the container with id 76505d04751a8ea3ed3c9a6a6173338dd60306a209ff2bfdb1f96b2bd24cf9b4 Oct 30 02:44:55.741220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:55.741189 5998 generic.go:356] "Generic (PLEG): container finished" podID="4f42c813-e505-4819-b8b5-f080cf92ffd6" containerID="d4936c4fb71ec7ce1b5fc815949b9fb7597bbc180a6752c3757a85e20a3e544e" exitCode=0 Oct 30 02:44:55.741343 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:55.741275 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp" event={"ID":"4f42c813-e505-4819-b8b5-f080cf92ffd6","Type":"ContainerDied","Data":"d4936c4fb71ec7ce1b5fc815949b9fb7597bbc180a6752c3757a85e20a3e544e"} Oct 30 02:44:56.342056 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:56.342014 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:56.751476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:56.751438 5998 generic.go:356] "Generic (PLEG): container finished" podID="c3508546-4215-44b4-9f02-4c90461c1083" containerID="bc99b9663c5d261f5fda83410fd5d8057491f5ab95ccaf447216eb139ecb1ef6" exitCode=0 Oct 30 02:44:56.751687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:56.751515 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x" event={"ID":"c3508546-4215-44b4-9f02-4c90461c1083","Type":"ContainerDied","Data":"bc99b9663c5d261f5fda83410fd5d8057491f5ab95ccaf447216eb139ecb1ef6"} Oct 30 02:44:56.751687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:56.751569 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x" event={"ID":"c3508546-4215-44b4-9f02-4c90461c1083","Type":"ContainerStarted","Data":"76505d04751a8ea3ed3c9a6a6173338dd60306a209ff2bfdb1f96b2bd24cf9b4"} Oct 30 02:44:56.903012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:56.902976 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp" Oct 30 02:44:56.994034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:56.993928 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f42c813-e505-4819-b8b5-f080cf92ffd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f42c813-e505-4819-b8b5-f080cf92ffd6-maas-api-openshift-ca\") pod \"4f42c813-e505-4819-b8b5-f080cf92ffd6\" (UID: \"4f42c813-e505-4819-b8b5-f080cf92ffd6\") " Oct 30 02:44:56.994034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:56.993990 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f42c813-e505-4819-b8b5-f080cf92ffd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f42c813-e505-4819-b8b5-f080cf92ffd6-maas-api-xks-ca\") pod \"4f42c813-e505-4819-b8b5-f080cf92ffd6\" (UID: \"4f42c813-e505-4819-b8b5-f080cf92ffd6\") " Oct 30 02:44:56.994034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:56.994024 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f42c813-e505-4819-b8b5-f080cf92ffd6-kube-api-access-kw7nz\" (UniqueName: \"kubernetes.io/projected/4f42c813-e505-4819-b8b5-f080cf92ffd6-kube-api-access-kw7nz\") pod \"4f42c813-e505-4819-b8b5-f080cf92ffd6\" (UID: \"4f42c813-e505-4819-b8b5-f080cf92ffd6\") " Oct 30 02:44:56.994419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:56.994337 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f42c813-e505-4819-b8b5-f080cf92ffd6-maas-api-openshift-ca" pod "4f42c813-e505-4819-b8b5-f080cf92ffd6" (UID: "4f42c813-e505-4819-b8b5-f080cf92ffd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:44:56.994720 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:56.994693 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f42c813-e505-4819-b8b5-f080cf92ffd6-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:44:56.996178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:56.996149 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f42c813-e505-4819-b8b5-f080cf92ffd6-maas-api-xks-ca" pod "4f42c813-e505-4819-b8b5-f080cf92ffd6" (UID: "4f42c813-e505-4819-b8b5-f080cf92ffd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:44:56.996282 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:56.996260 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f42c813-e505-4819-b8b5-f080cf92ffd6-kube-api-access-kw7nz" pod "4f42c813-e505-4819-b8b5-f080cf92ffd6" (UID: "4f42c813-e505-4819-b8b5-f080cf92ffd6"). InnerVolumeSpecName "kube-api-access-kw7nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:44:57.096032 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.095991 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f42c813-e505-4819-b8b5-f080cf92ffd6-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:44:57.096032 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.096029 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw7nz\" (UniqueName: \"kubernetes.io/projected/4f42c813-e505-4819-b8b5-f080cf92ffd6-kube-api-access-kw7nz\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:44:57.164263 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.164219 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m"] Oct 30 02:44:57.164859 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.164835 5998 cpu_manager.go:414] "containerMap: removing container" podUID="4f42c813-e505-4819-b8b5-f080cf92ffd6" containerName="revoke-keys" Oct 30 02:44:57.164962 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.164859 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f42c813-e505-4819-b8b5-f080cf92ffd6" containerName="revoke-keys" Oct 30 02:44:57.165043 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.165029 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f42c813-e505-4819-b8b5-f080cf92ffd6" containerName="revoke-keys" Oct 30 02:44:57.168547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.168522 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m" Oct 30 02:44:57.188527 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.188484 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m"] Oct 30 02:44:57.197182 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.197143 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85df47f5-8e2a-4eee-b6d8-3ec2636def64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c437787-dlr6m\" (UID: \"85df47f5-8e2a-4eee-b6d8-3ec2636def64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m" Oct 30 02:44:57.197347 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.197328 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85df47f5-8e2a-4eee-b6d8-3ec2636def64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c437787-dlr6m\" (UID: \"85df47f5-8e2a-4eee-b6d8-3ec2636def64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m" Oct 30 02:44:57.197394 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.197383 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shzgr\" (UniqueName: \"kubernetes.io/projected/85df47f5-8e2a-4eee-b6d8-3ec2636def64-kube-api-access-shzgr\") pod \"maas-api-revoke-sub-9c437787-dlr6m\" (UID: \"85df47f5-8e2a-4eee-b6d8-3ec2636def64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m" Oct 30 02:44:57.299137 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.298986 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85df47f5-8e2a-4eee-b6d8-3ec2636def64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c437787-dlr6m\" (UID: \"85df47f5-8e2a-4eee-b6d8-3ec2636def64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m" Oct 30 02:44:57.299357 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.299171 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85df47f5-8e2a-4eee-b6d8-3ec2636def64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c437787-dlr6m\" (UID: \"85df47f5-8e2a-4eee-b6d8-3ec2636def64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m" Oct 30 02:44:57.299357 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.299226 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shzgr\" (UniqueName: \"kubernetes.io/projected/85df47f5-8e2a-4eee-b6d8-3ec2636def64-kube-api-access-shzgr\") pod \"maas-api-revoke-sub-9c437787-dlr6m\" (UID: \"85df47f5-8e2a-4eee-b6d8-3ec2636def64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m" Oct 30 02:44:57.299828 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.299804 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85df47f5-8e2a-4eee-b6d8-3ec2636def64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c437787-dlr6m\" (UID: \"85df47f5-8e2a-4eee-b6d8-3ec2636def64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m" Oct 30 02:44:57.301577 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.301545 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85df47f5-8e2a-4eee-b6d8-3ec2636def64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c437787-dlr6m\" (UID: \"85df47f5-8e2a-4eee-b6d8-3ec2636def64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m" Oct 30 02:44:57.321125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.321071 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shzgr\" (UniqueName: \"kubernetes.io/projected/85df47f5-8e2a-4eee-b6d8-3ec2636def64-kube-api-access-shzgr\") pod \"maas-api-revoke-sub-9c437787-dlr6m\" (UID: \"85df47f5-8e2a-4eee-b6d8-3ec2636def64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m" Oct 30 02:44:57.482216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.482174 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m" Oct 30 02:44:57.759311 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.759279 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp" Oct 30 02:44:57.759500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.759304 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp" event={"ID":"4f42c813-e505-4819-b8b5-f080cf92ffd6","Type":"ContainerDied","Data":"8ce4fb72b3ae56fdcbd4248d3e4ac000ed2b38bcc54bbeafbda74bdd6af5c1de"} Oct 30 02:44:57.759500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.759361 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ce4fb72b3ae56fdcbd4248d3e4ac000ed2b38bcc54bbeafbda74bdd6af5c1de" Oct 30 02:44:57.789212 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:44:57.789183 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85df47f5_8e2a_4eee_b6d8_3ec2636def64.slice/crio-99ddd1e80dc5cf543515a48eeed22f7ef9962858b95f9f3bf366d8ad92fedbd4 WatchSource:0}: Error finding container 99ddd1e80dc5cf543515a48eeed22f7ef9962858b95f9f3bf366d8ad92fedbd4: Status 404 returned error can't find the container with id 99ddd1e80dc5cf543515a48eeed22f7ef9962858b95f9f3bf366d8ad92fedbd4 Oct 30 02:44:57.796528 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.796500 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m"] Oct 30 02:44:57.904248 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.904215 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x" Oct 30 02:44:57.912966 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:57.912930 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:58.009387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:58.009252 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3508546-4215-44b4-9f02-4c90461c1083-kube-api-access-h24wq\" (UniqueName: \"kubernetes.io/projected/c3508546-4215-44b4-9f02-4c90461c1083-kube-api-access-h24wq\") pod \"c3508546-4215-44b4-9f02-4c90461c1083\" (UID: \"c3508546-4215-44b4-9f02-4c90461c1083\") " Oct 30 02:44:58.009387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:58.009337 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3508546-4215-44b4-9f02-4c90461c1083-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3508546-4215-44b4-9f02-4c90461c1083-maas-api-openshift-ca\") pod \"c3508546-4215-44b4-9f02-4c90461c1083\" (UID: \"c3508546-4215-44b4-9f02-4c90461c1083\") " Oct 30 02:44:58.009626 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:58.009445 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3508546-4215-44b4-9f02-4c90461c1083-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3508546-4215-44b4-9f02-4c90461c1083-maas-api-xks-ca\") pod \"c3508546-4215-44b4-9f02-4c90461c1083\" (UID: \"c3508546-4215-44b4-9f02-4c90461c1083\") " Oct 30 02:44:58.010051 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:58.010012 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3508546-4215-44b4-9f02-4c90461c1083-maas-api-openshift-ca" pod "c3508546-4215-44b4-9f02-4c90461c1083" (UID: "c3508546-4215-44b4-9f02-4c90461c1083"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:44:58.012330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:58.012308 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3508546-4215-44b4-9f02-4c90461c1083-kube-api-access-h24wq" pod "c3508546-4215-44b4-9f02-4c90461c1083" (UID: "c3508546-4215-44b4-9f02-4c90461c1083"). InnerVolumeSpecName "kube-api-access-h24wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:44:58.013016 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:58.012998 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3508546-4215-44b4-9f02-4c90461c1083-maas-api-xks-ca" pod "c3508546-4215-44b4-9f02-4c90461c1083" (UID: "c3508546-4215-44b4-9f02-4c90461c1083"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:44:58.111678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:58.111638 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3508546-4215-44b4-9f02-4c90461c1083-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:44:58.111819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:58.111682 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h24wq\" (UniqueName: \"kubernetes.io/projected/c3508546-4215-44b4-9f02-4c90461c1083-kube-api-access-h24wq\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:44:58.111819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:58.111701 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3508546-4215-44b4-9f02-4c90461c1083-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:44:58.767834 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:58.767791 5998 generic.go:356] "Generic (PLEG): container finished" podID="85df47f5-8e2a-4eee-b6d8-3ec2636def64" containerID="a86e3538c48c1b8c7d1639f70a6902ede0fb06d530566567fddfc00823a09e1b" exitCode=0 Oct 30 02:44:58.768299 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:58.767880 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m" event={"ID":"85df47f5-8e2a-4eee-b6d8-3ec2636def64","Type":"ContainerDied","Data":"a86e3538c48c1b8c7d1639f70a6902ede0fb06d530566567fddfc00823a09e1b"} Oct 30 02:44:58.768299 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:58.767932 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m" event={"ID":"85df47f5-8e2a-4eee-b6d8-3ec2636def64","Type":"ContainerStarted","Data":"99ddd1e80dc5cf543515a48eeed22f7ef9962858b95f9f3bf366d8ad92fedbd4"} Oct 30 02:44:58.769454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:58.769434 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x" Oct 30 02:44:58.769499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:58.769463 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x" event={"ID":"c3508546-4215-44b4-9f02-4c90461c1083","Type":"ContainerDied","Data":"76505d04751a8ea3ed3c9a6a6173338dd60306a209ff2bfdb1f96b2bd24cf9b4"} Oct 30 02:44:58.769532 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:58.769503 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76505d04751a8ea3ed3c9a6a6173338dd60306a209ff2bfdb1f96b2bd24cf9b4" Oct 30 02:44:59.505322 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:59.505283 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:44:59.927795 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:44:59.927763 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m" Oct 30 02:45:00.040319 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.040224 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85df47f5-8e2a-4eee-b6d8-3ec2636def64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85df47f5-8e2a-4eee-b6d8-3ec2636def64-maas-api-openshift-ca\") pod \"85df47f5-8e2a-4eee-b6d8-3ec2636def64\" (UID: \"85df47f5-8e2a-4eee-b6d8-3ec2636def64\") " Oct 30 02:45:00.040487 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.040404 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85df47f5-8e2a-4eee-b6d8-3ec2636def64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85df47f5-8e2a-4eee-b6d8-3ec2636def64-maas-api-xks-ca\") pod \"85df47f5-8e2a-4eee-b6d8-3ec2636def64\" (UID: \"85df47f5-8e2a-4eee-b6d8-3ec2636def64\") " Oct 30 02:45:00.040487 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.040447 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85df47f5-8e2a-4eee-b6d8-3ec2636def64-kube-api-access-shzgr\" (UniqueName: \"kubernetes.io/projected/85df47f5-8e2a-4eee-b6d8-3ec2636def64-kube-api-access-shzgr\") pod \"85df47f5-8e2a-4eee-b6d8-3ec2636def64\" (UID: \"85df47f5-8e2a-4eee-b6d8-3ec2636def64\") " Oct 30 02:45:00.040741 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.040697 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85df47f5-8e2a-4eee-b6d8-3ec2636def64-maas-api-openshift-ca" pod "85df47f5-8e2a-4eee-b6d8-3ec2636def64" (UID: "85df47f5-8e2a-4eee-b6d8-3ec2636def64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:00.040988 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.040964 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85df47f5-8e2a-4eee-b6d8-3ec2636def64-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:00.043225 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.043194 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85df47f5-8e2a-4eee-b6d8-3ec2636def64-maas-api-xks-ca" pod "85df47f5-8e2a-4eee-b6d8-3ec2636def64" (UID: "85df47f5-8e2a-4eee-b6d8-3ec2636def64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:45:00.043322 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.043207 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85df47f5-8e2a-4eee-b6d8-3ec2636def64-kube-api-access-shzgr" pod "85df47f5-8e2a-4eee-b6d8-3ec2636def64" (UID: "85df47f5-8e2a-4eee-b6d8-3ec2636def64"). InnerVolumeSpecName "kube-api-access-shzgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:00.141810 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.141767 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85df47f5-8e2a-4eee-b6d8-3ec2636def64-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:00.142010 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.141804 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shzgr\" (UniqueName: \"kubernetes.io/projected/85df47f5-8e2a-4eee-b6d8-3ec2636def64-kube-api-access-shzgr\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:00.143691 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.143648 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld"] Oct 30 02:45:00.144587 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.144561 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c3508546-4215-44b4-9f02-4c90461c1083" containerName="revoke-keys" Oct 30 02:45:00.144646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.144592 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3508546-4215-44b4-9f02-4c90461c1083" containerName="revoke-keys" Oct 30 02:45:00.144646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.144615 5998 cpu_manager.go:414] "containerMap: removing container" podUID="85df47f5-8e2a-4eee-b6d8-3ec2636def64" containerName="revoke-keys" Oct 30 02:45:00.144646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.144626 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85df47f5-8e2a-4eee-b6d8-3ec2636def64" containerName="revoke-keys" Oct 30 02:45:00.144808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.144793 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3508546-4215-44b4-9f02-4c90461c1083" containerName="revoke-keys" Oct 30 02:45:00.144845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.144817 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="85df47f5-8e2a-4eee-b6d8-3ec2636def64" containerName="revoke-keys" Oct 30 02:45:00.148949 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.148916 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" Oct 30 02:45:00.151551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.151511 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2"] Oct 30 02:45:00.155335 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.155315 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" Oct 30 02:45:00.155767 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.155743 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v"] Oct 30 02:45:00.166707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.166669 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf"] Oct 30 02:45:00.172210 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.167726 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" Oct 30 02:45:00.173440 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.173413 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw"] Oct 30 02:45:00.173587 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.173563 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" Oct 30 02:45:00.177646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.177482 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" Oct 30 02:45:00.183059 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.183031 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld"] Oct 30 02:45:00.220043 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.220006 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2"] Oct 30 02:45:00.227025 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.226982 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7"] Oct 30 02:45:00.237327 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.237296 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v"] Oct 30 02:45:00.237516 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.237422 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" Oct 30 02:45:00.241561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.241535 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf"] Oct 30 02:45:00.243378 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.242641 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g7g5\" (UniqueName: \"kubernetes.io/projected/c560bec5-d0f1-4854-8ca0-24d44f73c750-kube-api-access-8g7g5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf\" (UID: \"c560bec5-d0f1-4854-8ca0-24d44f73c750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" Oct 30 02:45:00.243378 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.242704 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/929aedab-934e-4cb3-b540-3dc89c8a974a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw\" (UID: \"929aedab-934e-4cb3-b540-3dc89c8a974a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" Oct 30 02:45:00.243378 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.242734 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzdbh\" (UniqueName: \"kubernetes.io/projected/662d5231-266e-4ff9-8e5a-4409da2c7d82-kube-api-access-bzdbh\") pod \"maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v\" (UID: \"662d5231-266e-4ff9-8e5a-4409da2c7d82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" Oct 30 02:45:00.243378 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.242795 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gd2p\" (UniqueName: \"kubernetes.io/projected/c809da91-82e4-4c81-8644-d6409c5de91c-kube-api-access-8gd2p\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld\" (UID: \"c809da91-82e4-4c81-8644-d6409c5de91c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" Oct 30 02:45:00.243378 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.242828 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c560bec5-d0f1-4854-8ca0-24d44f73c750-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf\" (UID: \"c560bec5-d0f1-4854-8ca0-24d44f73c750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" Oct 30 02:45:00.243378 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.242884 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm6hl\" (UniqueName: \"kubernetes.io/projected/5bc5e6a0-5532-497a-8244-94b6768480f2-kube-api-access-tm6hl\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2\" (UID: \"5bc5e6a0-5532-497a-8244-94b6768480f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" Oct 30 02:45:00.243378 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.242999 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c809da91-82e4-4c81-8644-d6409c5de91c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld\" (UID: \"c809da91-82e4-4c81-8644-d6409c5de91c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" Oct 30 02:45:00.243378 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.243025 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xfs9\" (UniqueName: \"kubernetes.io/projected/929aedab-934e-4cb3-b540-3dc89c8a974a-kube-api-access-4xfs9\") pod \"maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw\" (UID: \"929aedab-934e-4cb3-b540-3dc89c8a974a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" Oct 30 02:45:00.243378 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.243103 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc5e6a0-5532-497a-8244-94b6768480f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2\" (UID: \"5bc5e6a0-5532-497a-8244-94b6768480f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" Oct 30 02:45:00.243378 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.243144 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/662d5231-266e-4ff9-8e5a-4409da2c7d82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v\" (UID: \"662d5231-266e-4ff9-8e5a-4409da2c7d82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" Oct 30 02:45:00.250650 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.248155 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw"] Oct 30 02:45:00.261078 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.261047 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2"] Oct 30 02:45:00.268278 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.268250 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw"] Oct 30 02:45:00.268434 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.268354 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" Oct 30 02:45:00.273177 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.273150 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" Oct 30 02:45:00.281393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.280396 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf"] Oct 30 02:45:00.284773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.284747 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" Oct 30 02:45:00.288251 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.287613 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7"] Oct 30 02:45:00.343106 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.343056 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g"] Oct 30 02:45:00.343999 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.343964 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf\" (UID: \"e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" Oct 30 02:45:00.344497 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.344423 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gd2p\" (UniqueName: \"kubernetes.io/projected/c809da91-82e4-4c81-8644-d6409c5de91c-kube-api-access-8gd2p\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld\" (UID: \"c809da91-82e4-4c81-8644-d6409c5de91c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" Oct 30 02:45:00.344604 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.344522 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0058da-63f6-48d6-9d96-316956c55dd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888805-sjxb2\" (UID: \"4a0058da-63f6-48d6-9d96-316956c55dd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" Oct 30 02:45:00.344604 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.344566 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c560bec5-d0f1-4854-8ca0-24d44f73c750-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf\" (UID: \"c560bec5-d0f1-4854-8ca0-24d44f73c750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" Oct 30 02:45:00.344725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.344618 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwgst\" (UniqueName: \"kubernetes.io/projected/b9f59989-527d-411d-9fc0-ce758b48e18a-kube-api-access-kwgst\") pod \"maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7\" (UID: \"b9f59989-527d-411d-9fc0-ce758b48e18a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" Oct 30 02:45:00.345471 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.345441 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c560bec5-d0f1-4854-8ca0-24d44f73c750-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf\" (UID: \"c560bec5-d0f1-4854-8ca0-24d44f73c750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" Oct 30 02:45:00.345563 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.345548 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6hl\" (UniqueName: \"kubernetes.io/projected/5bc5e6a0-5532-497a-8244-94b6768480f2-kube-api-access-tm6hl\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2\" (UID: \"5bc5e6a0-5532-497a-8244-94b6768480f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" Oct 30 02:45:00.345636 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.345616 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrpg4\" (UniqueName: \"kubernetes.io/projected/e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4-kube-api-access-rrpg4\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf\" (UID: \"e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" Oct 30 02:45:00.345699 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.345659 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrlqs\" (UniqueName: \"kubernetes.io/projected/8182c9bd-076d-4f26-8e5a-4596606d3d2d-kube-api-access-rrlqs\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw\" (UID: \"8182c9bd-076d-4f26-8e5a-4596606d3d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" Oct 30 02:45:00.345762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.345732 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drbmc\" (UniqueName: \"kubernetes.io/projected/4a0058da-63f6-48d6-9d96-316956c55dd4-kube-api-access-drbmc\") pod \"maas-api-key-cleanup-29888805-sjxb2\" (UID: \"4a0058da-63f6-48d6-9d96-316956c55dd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" Oct 30 02:45:00.345819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.345773 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c809da91-82e4-4c81-8644-d6409c5de91c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld\" (UID: \"c809da91-82e4-4c81-8644-d6409c5de91c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" Oct 30 02:45:00.345819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.345800 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xfs9\" (UniqueName: \"kubernetes.io/projected/929aedab-934e-4cb3-b540-3dc89c8a974a-kube-api-access-4xfs9\") pod \"maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw\" (UID: \"929aedab-934e-4cb3-b540-3dc89c8a974a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" Oct 30 02:45:00.345927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.345865 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc5e6a0-5532-497a-8244-94b6768480f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2\" (UID: \"5bc5e6a0-5532-497a-8244-94b6768480f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" Oct 30 02:45:00.345927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.345898 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/662d5231-266e-4ff9-8e5a-4409da2c7d82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v\" (UID: \"662d5231-266e-4ff9-8e5a-4409da2c7d82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" Oct 30 02:45:00.346029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.345983 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g7g5\" (UniqueName: \"kubernetes.io/projected/c560bec5-d0f1-4854-8ca0-24d44f73c750-kube-api-access-8g7g5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf\" (UID: \"c560bec5-d0f1-4854-8ca0-24d44f73c750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" Oct 30 02:45:00.346029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.346009 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8182c9bd-076d-4f26-8e5a-4596606d3d2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw\" (UID: \"8182c9bd-076d-4f26-8e5a-4596606d3d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" Oct 30 02:45:00.350784 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.346054 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/929aedab-934e-4cb3-b540-3dc89c8a974a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw\" (UID: \"929aedab-934e-4cb3-b540-3dc89c8a974a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" Oct 30 02:45:00.350784 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.346084 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzdbh\" (UniqueName: \"kubernetes.io/projected/662d5231-266e-4ff9-8e5a-4409da2c7d82-kube-api-access-bzdbh\") pod \"maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v\" (UID: \"662d5231-266e-4ff9-8e5a-4409da2c7d82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" Oct 30 02:45:00.350784 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.346128 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f59989-527d-411d-9fc0-ce758b48e18a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7\" (UID: \"b9f59989-527d-411d-9fc0-ce758b48e18a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" Oct 30 02:45:00.350784 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.346811 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c809da91-82e4-4c81-8644-d6409c5de91c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld\" (UID: \"c809da91-82e4-4c81-8644-d6409c5de91c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" Oct 30 02:45:00.350784 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.347194 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc5e6a0-5532-497a-8244-94b6768480f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2\" (UID: \"5bc5e6a0-5532-497a-8244-94b6768480f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" Oct 30 02:45:00.350784 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.347775 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/662d5231-266e-4ff9-8e5a-4409da2c7d82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v\" (UID: \"662d5231-266e-4ff9-8e5a-4409da2c7d82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" Oct 30 02:45:00.350784 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.347879 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/929aedab-934e-4cb3-b540-3dc89c8a974a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw\" (UID: \"929aedab-934e-4cb3-b540-3dc89c8a974a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" Oct 30 02:45:00.365959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.365926 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw"] Oct 30 02:45:00.366213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.366192 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2"] Oct 30 02:45:00.366386 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.366364 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf"] Oct 30 02:45:00.366522 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.366236 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" Oct 30 02:45:00.371673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.371645 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g"] Oct 30 02:45:00.387162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.380662 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g7g5\" (UniqueName: \"kubernetes.io/projected/c560bec5-d0f1-4854-8ca0-24d44f73c750-kube-api-access-8g7g5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf\" (UID: \"c560bec5-d0f1-4854-8ca0-24d44f73c750\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" Oct 30 02:45:00.387162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.381829 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzdbh\" (UniqueName: \"kubernetes.io/projected/662d5231-266e-4ff9-8e5a-4409da2c7d82-kube-api-access-bzdbh\") pod \"maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v\" (UID: \"662d5231-266e-4ff9-8e5a-4409da2c7d82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" Oct 30 02:45:00.387162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.382468 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xfs9\" (UniqueName: \"kubernetes.io/projected/929aedab-934e-4cb3-b540-3dc89c8a974a-kube-api-access-4xfs9\") pod \"maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw\" (UID: \"929aedab-934e-4cb3-b540-3dc89c8a974a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" Oct 30 02:45:00.387162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.383735 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm6hl\" (UniqueName: \"kubernetes.io/projected/5bc5e6a0-5532-497a-8244-94b6768480f2-kube-api-access-tm6hl\") pod \"maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2\" (UID: \"5bc5e6a0-5532-497a-8244-94b6768480f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" Oct 30 02:45:00.387162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.385982 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gd2p\" (UniqueName: \"kubernetes.io/projected/c809da91-82e4-4c81-8644-d6409c5de91c-kube-api-access-8gd2p\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld\" (UID: \"c809da91-82e4-4c81-8644-d6409c5de91c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" Oct 30 02:45:00.447874 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.447827 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrpg4\" (UniqueName: \"kubernetes.io/projected/e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4-kube-api-access-rrpg4\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf\" (UID: \"e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" Oct 30 02:45:00.448084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.447905 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrlqs\" (UniqueName: \"kubernetes.io/projected/8182c9bd-076d-4f26-8e5a-4596606d3d2d-kube-api-access-rrlqs\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw\" (UID: \"8182c9bd-076d-4f26-8e5a-4596606d3d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" Oct 30 02:45:00.448084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.447953 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9dvp\" (UniqueName: \"kubernetes.io/projected/cce212c5-9e99-4e58-8dde-89b23f8e9e2e-kube-api-access-m9dvp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g\" (UID: \"cce212c5-9e99-4e58-8dde-89b23f8e9e2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" Oct 30 02:45:00.448084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.448014 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drbmc\" (UniqueName: \"kubernetes.io/projected/4a0058da-63f6-48d6-9d96-316956c55dd4-kube-api-access-drbmc\") pod \"maas-api-key-cleanup-29888805-sjxb2\" (UID: \"4a0058da-63f6-48d6-9d96-316956c55dd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" Oct 30 02:45:00.448338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.448164 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8182c9bd-076d-4f26-8e5a-4596606d3d2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw\" (UID: \"8182c9bd-076d-4f26-8e5a-4596606d3d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" Oct 30 02:45:00.448338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.448226 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce212c5-9e99-4e58-8dde-89b23f8e9e2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g\" (UID: \"cce212c5-9e99-4e58-8dde-89b23f8e9e2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" Oct 30 02:45:00.448338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.448288 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f59989-527d-411d-9fc0-ce758b48e18a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7\" (UID: \"b9f59989-527d-411d-9fc0-ce758b48e18a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" Oct 30 02:45:00.448507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.448340 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf\" (UID: \"e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" Oct 30 02:45:00.448507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.448375 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0058da-63f6-48d6-9d96-316956c55dd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888805-sjxb2\" (UID: \"4a0058da-63f6-48d6-9d96-316956c55dd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" Oct 30 02:45:00.448507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.448433 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwgst\" (UniqueName: \"kubernetes.io/projected/b9f59989-527d-411d-9fc0-ce758b48e18a-kube-api-access-kwgst\") pod \"maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7\" (UID: \"b9f59989-527d-411d-9fc0-ce758b48e18a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" Oct 30 02:45:00.448964 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.448935 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8182c9bd-076d-4f26-8e5a-4596606d3d2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw\" (UID: \"8182c9bd-076d-4f26-8e5a-4596606d3d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" Oct 30 02:45:00.449381 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.449357 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f59989-527d-411d-9fc0-ce758b48e18a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7\" (UID: \"b9f59989-527d-411d-9fc0-ce758b48e18a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" Oct 30 02:45:00.449381 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.449369 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf\" (UID: \"e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" Oct 30 02:45:00.449526 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.449497 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0058da-63f6-48d6-9d96-316956c55dd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888805-sjxb2\" (UID: \"4a0058da-63f6-48d6-9d96-316956c55dd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" Oct 30 02:45:00.471201 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.471170 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwgst\" (UniqueName: \"kubernetes.io/projected/b9f59989-527d-411d-9fc0-ce758b48e18a-kube-api-access-kwgst\") pod \"maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7\" (UID: \"b9f59989-527d-411d-9fc0-ce758b48e18a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" Oct 30 02:45:00.471201 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.471186 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrpg4\" (UniqueName: \"kubernetes.io/projected/e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4-kube-api-access-rrpg4\") pod \"maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf\" (UID: \"e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" Oct 30 02:45:00.471390 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.471177 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drbmc\" (UniqueName: \"kubernetes.io/projected/4a0058da-63f6-48d6-9d96-316956c55dd4-kube-api-access-drbmc\") pod \"maas-api-key-cleanup-29888805-sjxb2\" (UID: \"4a0058da-63f6-48d6-9d96-316956c55dd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" Oct 30 02:45:00.471390 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.471293 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrlqs\" (UniqueName: \"kubernetes.io/projected/8182c9bd-076d-4f26-8e5a-4596606d3d2d-kube-api-access-rrlqs\") pod \"maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw\" (UID: \"8182c9bd-076d-4f26-8e5a-4596606d3d2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" Oct 30 02:45:00.477049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.477032 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" Oct 30 02:45:00.487562 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.487538 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" Oct 30 02:45:00.498618 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.498588 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" Oct 30 02:45:00.510529 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.510497 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" Oct 30 02:45:00.534849 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.534589 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" Oct 30 02:45:00.550316 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.550274 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce212c5-9e99-4e58-8dde-89b23f8e9e2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g\" (UID: \"cce212c5-9e99-4e58-8dde-89b23f8e9e2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" Oct 30 02:45:00.550491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.550468 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9dvp\" (UniqueName: \"kubernetes.io/projected/cce212c5-9e99-4e58-8dde-89b23f8e9e2e-kube-api-access-m9dvp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g\" (UID: \"cce212c5-9e99-4e58-8dde-89b23f8e9e2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" Oct 30 02:45:00.559274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.558012 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" Oct 30 02:45:00.559274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.558602 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce212c5-9e99-4e58-8dde-89b23f8e9e2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g\" (UID: \"cce212c5-9e99-4e58-8dde-89b23f8e9e2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" Oct 30 02:45:00.595597 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.595555 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" Oct 30 02:45:00.603637 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.603390 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" Oct 30 02:45:00.611707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.611440 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" Oct 30 02:45:00.620320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.618792 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9dvp\" (UniqueName: \"kubernetes.io/projected/cce212c5-9e99-4e58-8dde-89b23f8e9e2e-kube-api-access-m9dvp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g\" (UID: \"cce212c5-9e99-4e58-8dde-89b23f8e9e2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" Oct 30 02:45:00.722841 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.722608 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" Oct 30 02:45:00.799351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.798703 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m" event={"ID":"85df47f5-8e2a-4eee-b6d8-3ec2636def64","Type":"ContainerDied","Data":"99ddd1e80dc5cf543515a48eeed22f7ef9962858b95f9f3bf366d8ad92fedbd4"} Oct 30 02:45:00.799351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.798753 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99ddd1e80dc5cf543515a48eeed22f7ef9962858b95f9f3bf366d8ad92fedbd4" Oct 30 02:45:00.799351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.798808 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m" Oct 30 02:45:00.819798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:00.817399 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld"] Oct 30 02:45:01.025939 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.025888 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2"] Oct 30 02:45:01.077513 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.077419 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v"] Oct 30 02:45:01.081045 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.081017 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf"] Oct 30 02:45:01.083627 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:45:01.083597 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662d5231_266e_4ff9_8e5a_4409da2c7d82.slice/crio-fc8167c14cdad5cf51fd2fffd318df6d6aa211717921b24a096d5b52ce62d102 WatchSource:0}: Error finding container fc8167c14cdad5cf51fd2fffd318df6d6aa211717921b24a096d5b52ce62d102: Status 404 returned error can't find the container with id fc8167c14cdad5cf51fd2fffd318df6d6aa211717921b24a096d5b52ce62d102 Oct 30 02:45:01.085320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.085294 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2"] Oct 30 02:45:01.085959 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:45:01.085936 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc560bec5_d0f1_4854_8ca0_24d44f73c750.slice/crio-c939db2a4a3c585a0811a88a3113eae677027ce569bc39e7a1f8cb34c84d4b33 WatchSource:0}: Error finding container c939db2a4a3c585a0811a88a3113eae677027ce569bc39e7a1f8cb34c84d4b33: Status 404 returned error can't find the container with id c939db2a4a3c585a0811a88a3113eae677027ce569bc39e7a1f8cb34c84d4b33 Oct 30 02:45:01.086317 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:45:01.086291 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0058da_63f6_48d6_9d96_316956c55dd4.slice/crio-5c8962ee9bace5e0c0724e3ad3e1162f9f54b9d5e0959f66479efca2b3f1eca1 WatchSource:0}: Error finding container 5c8962ee9bace5e0c0724e3ad3e1162f9f54b9d5e0959f66479efca2b3f1eca1: Status 404 returned error can't find the container with id 5c8962ee9bace5e0c0724e3ad3e1162f9f54b9d5e0959f66479efca2b3f1eca1 Oct 30 02:45:01.105933 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.105899 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g"] Oct 30 02:45:01.116752 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.116717 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw"] Oct 30 02:45:01.133944 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.133910 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7"] Oct 30 02:45:01.194550 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.194512 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:01.205653 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:45:01.205613 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce212c5_9e99_4e58_8dde_89b23f8e9e2e.slice/crio-0033f12a0f303a9f459236ce4c1cd6073714ecd3c2d688c8d682ecae02d473b7 WatchSource:0}: Error finding container 0033f12a0f303a9f459236ce4c1cd6073714ecd3c2d688c8d682ecae02d473b7: Status 404 returned error can't find the container with id 0033f12a0f303a9f459236ce4c1cd6073714ecd3c2d688c8d682ecae02d473b7 Oct 30 02:45:01.226680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.226645 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw"] Oct 30 02:45:01.267133 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.265742 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf"] Oct 30 02:45:01.815393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.815301 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" event={"ID":"929aedab-934e-4cb3-b540-3dc89c8a974a","Type":"ContainerStarted","Data":"1b409615a46652a41e8587066dbac4944826aab4415815fa5ce85c9a97435219"} Oct 30 02:45:01.818641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.818609 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" event={"ID":"8182c9bd-076d-4f26-8e5a-4596606d3d2d","Type":"ContainerStarted","Data":"a35983d4a607b39529bf6e1144d808a970c01655d22b00d67b17a5eda7a3715b"} Oct 30 02:45:01.822250 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.821955 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" event={"ID":"e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4","Type":"ContainerStarted","Data":"593d4e4b00b16f592deca6b9e34808ac28281dd00d10aff17b10c497b8639479"} Oct 30 02:45:01.823730 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.823703 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" event={"ID":"662d5231-266e-4ff9-8e5a-4409da2c7d82","Type":"ContainerStarted","Data":"fc8167c14cdad5cf51fd2fffd318df6d6aa211717921b24a096d5b52ce62d102"} Oct 30 02:45:01.827609 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.827582 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" event={"ID":"4a0058da-63f6-48d6-9d96-316956c55dd4","Type":"ContainerStarted","Data":"15b07948a17216e633f36828ab2b00cf4a39931d16b4768f5bebb0b337455c6b"} Oct 30 02:45:01.827717 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.827634 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" event={"ID":"4a0058da-63f6-48d6-9d96-316956c55dd4","Type":"ContainerStarted","Data":"5c8962ee9bace5e0c0724e3ad3e1162f9f54b9d5e0959f66479efca2b3f1eca1"} Oct 30 02:45:01.830586 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.830560 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" event={"ID":"b9f59989-527d-411d-9fc0-ce758b48e18a","Type":"ContainerStarted","Data":"7a73e0cbeeedb9a643b486133a3b3ad15581d513802d17e50599d967a05260e4"} Oct 30 02:45:01.832340 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.832314 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" event={"ID":"cce212c5-9e99-4e58-8dde-89b23f8e9e2e","Type":"ContainerStarted","Data":"0033f12a0f303a9f459236ce4c1cd6073714ecd3c2d688c8d682ecae02d473b7"} Oct 30 02:45:01.834195 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.834159 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" event={"ID":"c809da91-82e4-4c81-8644-d6409c5de91c","Type":"ContainerStarted","Data":"b549850223a8e5052a5a96c2d8d9573323b469f031fe7b1ff45c14d6fb01226a"} Oct 30 02:45:01.834550 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.834211 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" event={"ID":"c809da91-82e4-4c81-8644-d6409c5de91c","Type":"ContainerStarted","Data":"631b0057a3116da442f9d9b88cbfc9f2250d9e61a30f00e3c0809f02e90d6152"} Oct 30 02:45:01.836973 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.836943 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" event={"ID":"5bc5e6a0-5532-497a-8244-94b6768480f2","Type":"ContainerStarted","Data":"df33e81cfc287377c09d73dfb081d6547ecb81f229159fedaaa89323171b1cdc"} Oct 30 02:45:01.837103 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.836994 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" event={"ID":"5bc5e6a0-5532-497a-8244-94b6768480f2","Type":"ContainerStarted","Data":"a403dd30739d9888039678ecbff33f9da936aefc6a5e237700b97fb76aaab654"} Oct 30 02:45:01.838812 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.838791 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" event={"ID":"c560bec5-d0f1-4854-8ca0-24d44f73c750","Type":"ContainerStarted","Data":"c939db2a4a3c585a0811a88a3113eae677027ce569bc39e7a1f8cb34c84d4b33"} Oct 30 02:45:01.867623 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.867559 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" podStartSLOduration=1.8675447269999998 podStartE2EDuration="1.867544727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:01.864264179 +0000 UTC m=+4531.443055260" watchObservedRunningTime="2026-10-30 02:45:01.867544727 +0000 UTC m=+4531.446335844" Oct 30 02:45:01.907105 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.906877 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" podStartSLOduration=1.9068564289999999 podStartE2EDuration="1.906856429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:01.903050547 +0000 UTC m=+4531.481841628" watchObservedRunningTime="2026-10-30 02:45:01.906856429 +0000 UTC m=+4531.485647511" Oct 30 02:45:01.943385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.943327 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" podStartSLOduration=1.943311805 podStartE2EDuration="1.943311805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:01.938794097 +0000 UTC m=+4531.517585182" watchObservedRunningTime="2026-10-30 02:45:01.943311805 +0000 UTC m=+4531.522102885" Oct 30 02:45:01.978653 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:01.978578 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" podStartSLOduration=1.9785572390000001 podStartE2EDuration="1.978557239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:01.976257841 +0000 UTC m=+4531.555048922" watchObservedRunningTime="2026-10-30 02:45:01.978557239 +0000 UTC m=+4531.557348386" Oct 30 02:45:02.746534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.746488 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:02.850900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.850855 5998 generic.go:356] "Generic (PLEG): container finished" podID="c809da91-82e4-4c81-8644-d6409c5de91c" containerID="b549850223a8e5052a5a96c2d8d9573323b469f031fe7b1ff45c14d6fb01226a" exitCode=60 Oct 30 02:45:02.851102 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.850946 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" event={"ID":"c809da91-82e4-4c81-8644-d6409c5de91c","Type":"ContainerDied","Data":"b549850223a8e5052a5a96c2d8d9573323b469f031fe7b1ff45c14d6fb01226a"} Oct 30 02:45:02.851504 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.851483 5998 scope.go:122] "RemoveContainer" containerID="b549850223a8e5052a5a96c2d8d9573323b469f031fe7b1ff45c14d6fb01226a" Oct 30 02:45:02.856509 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.855811 5998 generic.go:356] "Generic (PLEG): container finished" podID="5bc5e6a0-5532-497a-8244-94b6768480f2" containerID="df33e81cfc287377c09d73dfb081d6547ecb81f229159fedaaa89323171b1cdc" exitCode=60 Oct 30 02:45:02.856509 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.855910 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" event={"ID":"5bc5e6a0-5532-497a-8244-94b6768480f2","Type":"ContainerDied","Data":"df33e81cfc287377c09d73dfb081d6547ecb81f229159fedaaa89323171b1cdc"} Oct 30 02:45:02.856509 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.856221 5998 scope.go:122] "RemoveContainer" containerID="df33e81cfc287377c09d73dfb081d6547ecb81f229159fedaaa89323171b1cdc" Oct 30 02:45:02.862815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.861716 5998 generic.go:356] "Generic (PLEG): container finished" podID="c560bec5-d0f1-4854-8ca0-24d44f73c750" containerID="d6830cd9274bb9e449393bfab9ef5ddef14ffd2b173418d59e0946e0fbff2815" exitCode=60 Oct 30 02:45:02.862815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.861810 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" event={"ID":"c560bec5-d0f1-4854-8ca0-24d44f73c750","Type":"ContainerDied","Data":"d6830cd9274bb9e449393bfab9ef5ddef14ffd2b173418d59e0946e0fbff2815"} Oct 30 02:45:02.862815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.862165 5998 scope.go:122] "RemoveContainer" containerID="d6830cd9274bb9e449393bfab9ef5ddef14ffd2b173418d59e0946e0fbff2815" Oct 30 02:45:02.863815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.863789 5998 generic.go:356] "Generic (PLEG): container finished" podID="929aedab-934e-4cb3-b540-3dc89c8a974a" containerID="b624a2e6948f3a71d805ee48926a4b3aa38b59681c2e0e3ba2aff839e213ad67" exitCode=60 Oct 30 02:45:02.863942 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.863866 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" event={"ID":"929aedab-934e-4cb3-b540-3dc89c8a974a","Type":"ContainerDied","Data":"b624a2e6948f3a71d805ee48926a4b3aa38b59681c2e0e3ba2aff839e213ad67"} Oct 30 02:45:02.864226 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.864161 5998 scope.go:122] "RemoveContainer" containerID="b624a2e6948f3a71d805ee48926a4b3aa38b59681c2e0e3ba2aff839e213ad67" Oct 30 02:45:02.866033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.865682 5998 generic.go:356] "Generic (PLEG): container finished" podID="8182c9bd-076d-4f26-8e5a-4596606d3d2d" containerID="931e874b00c3a16f26e0bea073ea498eb730e7bc3980dfd539ba0268ada15120" exitCode=60 Oct 30 02:45:02.866033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.865766 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" event={"ID":"8182c9bd-076d-4f26-8e5a-4596606d3d2d","Type":"ContainerDied","Data":"931e874b00c3a16f26e0bea073ea498eb730e7bc3980dfd539ba0268ada15120"} Oct 30 02:45:02.866033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.865965 5998 scope.go:122] "RemoveContainer" containerID="931e874b00c3a16f26e0bea073ea498eb730e7bc3980dfd539ba0268ada15120" Oct 30 02:45:02.868736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.868614 5998 generic.go:356] "Generic (PLEG): container finished" podID="e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4" containerID="a0819d89b13b85659159b05cb0186f2a4e941f7c9a37f0f8619fc06e4dc74031" exitCode=60 Oct 30 02:45:02.868736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.868723 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" event={"ID":"e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4","Type":"ContainerDied","Data":"a0819d89b13b85659159b05cb0186f2a4e941f7c9a37f0f8619fc06e4dc74031"} Oct 30 02:45:02.868911 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.868895 5998 scope.go:122] "RemoveContainer" containerID="a0819d89b13b85659159b05cb0186f2a4e941f7c9a37f0f8619fc06e4dc74031" Oct 30 02:45:02.870385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.870365 5998 generic.go:356] "Generic (PLEG): container finished" podID="662d5231-266e-4ff9-8e5a-4409da2c7d82" containerID="ca42027cd31464972c0146f006c4e30489c5d9e39a8b3d08790d0a8d02a79913" exitCode=60 Oct 30 02:45:02.870488 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.870443 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" event={"ID":"662d5231-266e-4ff9-8e5a-4409da2c7d82","Type":"ContainerDied","Data":"ca42027cd31464972c0146f006c4e30489c5d9e39a8b3d08790d0a8d02a79913"} Oct 30 02:45:02.870627 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.870613 5998 scope.go:122] "RemoveContainer" containerID="ca42027cd31464972c0146f006c4e30489c5d9e39a8b3d08790d0a8d02a79913" Oct 30 02:45:02.872058 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.872040 5998 generic.go:356] "Generic (PLEG): container finished" podID="4a0058da-63f6-48d6-9d96-316956c55dd4" containerID="15b07948a17216e633f36828ab2b00cf4a39931d16b4768f5bebb0b337455c6b" exitCode=60 Oct 30 02:45:02.872178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.872073 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" event={"ID":"4a0058da-63f6-48d6-9d96-316956c55dd4","Type":"ContainerDied","Data":"15b07948a17216e633f36828ab2b00cf4a39931d16b4768f5bebb0b337455c6b"} Oct 30 02:45:02.872363 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.872345 5998 scope.go:122] "RemoveContainer" containerID="15b07948a17216e633f36828ab2b00cf4a39931d16b4768f5bebb0b337455c6b" Oct 30 02:45:02.873863 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.873795 5998 generic.go:356] "Generic (PLEG): container finished" podID="b9f59989-527d-411d-9fc0-ce758b48e18a" containerID="9ddc12ef9fbe1c7275f0292dc582c1d2502eb69f5988e67321185d55e8cc1ae2" exitCode=60 Oct 30 02:45:02.873924 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.873880 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" event={"ID":"b9f59989-527d-411d-9fc0-ce758b48e18a","Type":"ContainerDied","Data":"9ddc12ef9fbe1c7275f0292dc582c1d2502eb69f5988e67321185d55e8cc1ae2"} Oct 30 02:45:02.874040 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.874026 5998 scope.go:122] "RemoveContainer" containerID="9ddc12ef9fbe1c7275f0292dc582c1d2502eb69f5988e67321185d55e8cc1ae2" Oct 30 02:45:02.876908 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.876619 5998 generic.go:356] "Generic (PLEG): container finished" podID="cce212c5-9e99-4e58-8dde-89b23f8e9e2e" containerID="a5a1f5fc5b129c4ca8598cc363edcbbca9a5fb2594908c17dfa4bab65d4cfffc" exitCode=60 Oct 30 02:45:02.876908 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.876662 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" event={"ID":"cce212c5-9e99-4e58-8dde-89b23f8e9e2e","Type":"ContainerDied","Data":"a5a1f5fc5b129c4ca8598cc363edcbbca9a5fb2594908c17dfa4bab65d4cfffc"} Oct 30 02:45:02.877212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:02.876921 5998 scope.go:122] "RemoveContainer" containerID="a5a1f5fc5b129c4ca8598cc363edcbbca9a5fb2594908c17dfa4bab65d4cfffc" Oct 30 02:45:03.886975 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:03.886878 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" event={"ID":"8182c9bd-076d-4f26-8e5a-4596606d3d2d","Type":"ContainerStarted","Data":"8017f7561ba2e60dc781cc4422748b8975c147895a6b02eb595ff1a6736ef822"} Oct 30 02:45:03.889046 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:03.889014 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" event={"ID":"e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4","Type":"ContainerStarted","Data":"a4a0fb458be0d3a6b59f584c9f0d7290502b0d64ba770b4b8d3286b0b5d3f7eb"} Oct 30 02:45:03.891003 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:03.890972 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" event={"ID":"662d5231-266e-4ff9-8e5a-4409da2c7d82","Type":"ContainerStarted","Data":"6b7d910316d04683370686f4148be9d21d5f1b679e74bb79dc9f2fc295a5c7ed"} Oct 30 02:45:03.892935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:03.892908 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" event={"ID":"4a0058da-63f6-48d6-9d96-316956c55dd4","Type":"ContainerStarted","Data":"a85aa7897284fe1a3932c5d7d34a4e1d603c215a36943f3ef1fc3b2cd41375e0"} Oct 30 02:45:03.895023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:03.894996 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" event={"ID":"b9f59989-527d-411d-9fc0-ce758b48e18a","Type":"ContainerStarted","Data":"82799175b0f56c945563f8c56a785f758fcc45b4c9a0e918005b9e269ced7a99"} Oct 30 02:45:03.897289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:03.897264 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" event={"ID":"cce212c5-9e99-4e58-8dde-89b23f8e9e2e","Type":"ContainerStarted","Data":"75e97c06d52cbf33f9ef64a178856f81ceb2c5033b03f3754c517606e0ade35c"} Oct 30 02:45:03.899734 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:03.899708 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" event={"ID":"c809da91-82e4-4c81-8644-d6409c5de91c","Type":"ContainerStarted","Data":"2b5b0dd77b0c419d53bdb8e5327097202e1f21047be211ea6e01377104b0766a"} Oct 30 02:45:03.907251 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:03.907220 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" event={"ID":"5bc5e6a0-5532-497a-8244-94b6768480f2","Type":"ContainerStarted","Data":"8aaeb6f825f64a64ff0a9592aefd973b03e1b38c1347cd4d8f5f8a1edfe935d0"} Oct 30 02:45:03.910426 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:03.910345 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" event={"ID":"c560bec5-d0f1-4854-8ca0-24d44f73c750","Type":"ContainerStarted","Data":"3d117c0adf46209225d6ae72d3b8e98a942193c1a26fb7d57caf87ec81c9deb8"} Oct 30 02:45:03.912463 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:03.912433 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" event={"ID":"929aedab-934e-4cb3-b540-3dc89c8a974a","Type":"ContainerStarted","Data":"d90b664cc9ee653df11ecd82c4592cac0a415415188a0cfc126328c811c6b34b"} Oct 30 02:45:03.927367 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:03.927311 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" podStartSLOduration=3.927296502 podStartE2EDuration="3.927296502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:03.923562717 +0000 UTC m=+4533.502353799" watchObservedRunningTime="2026-10-30 02:45:03.927296502 +0000 UTC m=+4533.506087582" Oct 30 02:45:03.969169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:03.969110 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" podStartSLOduration=3.969079575 podStartE2EDuration="3.969079575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:03.96471375 +0000 UTC m=+4533.543504830" watchObservedRunningTime="2026-10-30 02:45:03.969079575 +0000 UTC m=+4533.547870676" Oct 30 02:45:04.050852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.050759 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" podStartSLOduration=4.050740167 podStartE2EDuration="4.050740167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:04.048272775 +0000 UTC m=+4533.627063855" watchObservedRunningTime="2026-10-30 02:45:04.050740167 +0000 UTC m=+4533.629531250" Oct 30 02:45:04.211949 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.211786 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" podStartSLOduration=4.211710563 podStartE2EDuration="4.211710563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:04.205788557 +0000 UTC m=+4533.784579639" watchObservedRunningTime="2026-10-30 02:45:04.211710563 +0000 UTC m=+4533.790501646" Oct 30 02:45:04.244079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.244025 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" podStartSLOduration=4.244006797 podStartE2EDuration="4.244006797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:04.242919229 +0000 UTC m=+4533.821710311" watchObservedRunningTime="2026-10-30 02:45:04.244006797 +0000 UTC m=+4533.822797931" Oct 30 02:45:04.284375 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.284318 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" podStartSLOduration=4.284305593 podStartE2EDuration="4.284305593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:04.282707812 +0000 UTC m=+4533.861498903" watchObservedRunningTime="2026-10-30 02:45:04.284305593 +0000 UTC m=+4533.863096688" Oct 30 02:45:04.421511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.421478 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:04.421606 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.421521 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc"] Oct 30 02:45:04.426692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.426667 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc" Oct 30 02:45:04.444348 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.444310 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc"] Oct 30 02:45:04.504084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.504028 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twjgj\" (UniqueName: \"kubernetes.io/projected/e1d4b059-e455-4127-bccc-da21bf6e0680-kube-api-access-twjgj\") pod \"maas-api-revoke-sub-5bb3ae5d-drprc\" (UID: \"e1d4b059-e455-4127-bccc-da21bf6e0680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc" Oct 30 02:45:04.504447 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.504315 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d4b059-e455-4127-bccc-da21bf6e0680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bb3ae5d-drprc\" (UID: \"e1d4b059-e455-4127-bccc-da21bf6e0680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc" Oct 30 02:45:04.504447 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.504357 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d4b059-e455-4127-bccc-da21bf6e0680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bb3ae5d-drprc\" (UID: \"e1d4b059-e455-4127-bccc-da21bf6e0680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc" Oct 30 02:45:04.509278 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.509251 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:45:04.605897 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.605851 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twjgj\" (UniqueName: \"kubernetes.io/projected/e1d4b059-e455-4127-bccc-da21bf6e0680-kube-api-access-twjgj\") pod \"maas-api-revoke-sub-5bb3ae5d-drprc\" (UID: \"e1d4b059-e455-4127-bccc-da21bf6e0680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc" Oct 30 02:45:04.606205 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.606179 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d4b059-e455-4127-bccc-da21bf6e0680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bb3ae5d-drprc\" (UID: \"e1d4b059-e455-4127-bccc-da21bf6e0680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc" Oct 30 02:45:04.606333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.606235 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d4b059-e455-4127-bccc-da21bf6e0680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bb3ae5d-drprc\" (UID: \"e1d4b059-e455-4127-bccc-da21bf6e0680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc" Oct 30 02:45:04.606950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.606931 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d4b059-e455-4127-bccc-da21bf6e0680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bb3ae5d-drprc\" (UID: \"e1d4b059-e455-4127-bccc-da21bf6e0680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc" Oct 30 02:45:04.609559 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.609540 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d4b059-e455-4127-bccc-da21bf6e0680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bb3ae5d-drprc\" (UID: \"e1d4b059-e455-4127-bccc-da21bf6e0680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc" Oct 30 02:45:04.630495 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.630461 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twjgj\" (UniqueName: \"kubernetes.io/projected/e1d4b059-e455-4127-bccc-da21bf6e0680-kube-api-access-twjgj\") pod \"maas-api-revoke-sub-5bb3ae5d-drprc\" (UID: \"e1d4b059-e455-4127-bccc-da21bf6e0680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc" Oct 30 02:45:04.756912 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.756880 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc" Oct 30 02:45:04.924445 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.924404 5998 generic.go:356] "Generic (PLEG): container finished" podID="e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4" containerID="a4a0fb458be0d3a6b59f584c9f0d7290502b0d64ba770b4b8d3286b0b5d3f7eb" exitCode=60 Oct 30 02:45:04.924922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.924481 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" event={"ID":"e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4","Type":"ContainerDied","Data":"a4a0fb458be0d3a6b59f584c9f0d7290502b0d64ba770b4b8d3286b0b5d3f7eb"} Oct 30 02:45:04.924922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.924567 5998 scope.go:122] "RemoveContainer" containerID="a0819d89b13b85659159b05cb0186f2a4e941f7c9a37f0f8619fc06e4dc74031" Oct 30 02:45:04.924922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.924808 5998 scope.go:122] "RemoveContainer" containerID="a4a0fb458be0d3a6b59f584c9f0d7290502b0d64ba770b4b8d3286b0b5d3f7eb" Oct 30 02:45:04.925214 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:04.925186 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf_odh-ai-gateway-infra(e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" podUID="e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4" Oct 30 02:45:04.926619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.926597 5998 generic.go:356] "Generic (PLEG): container finished" podID="662d5231-266e-4ff9-8e5a-4409da2c7d82" containerID="6b7d910316d04683370686f4148be9d21d5f1b679e74bb79dc9f2fc295a5c7ed" exitCode=60 Oct 30 02:45:04.926729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.926664 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" event={"ID":"662d5231-266e-4ff9-8e5a-4409da2c7d82","Type":"ContainerDied","Data":"6b7d910316d04683370686f4148be9d21d5f1b679e74bb79dc9f2fc295a5c7ed"} Oct 30 02:45:04.926959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.926943 5998 scope.go:122] "RemoveContainer" containerID="6b7d910316d04683370686f4148be9d21d5f1b679e74bb79dc9f2fc295a5c7ed" Oct 30 02:45:04.927300 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:04.927276 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v_odh-ai-gateway-infra(662d5231-266e-4ff9-8e5a-4409da2c7d82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" podUID="662d5231-266e-4ff9-8e5a-4409da2c7d82" Oct 30 02:45:04.928482 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.928458 5998 generic.go:356] "Generic (PLEG): container finished" podID="4a0058da-63f6-48d6-9d96-316956c55dd4" containerID="a85aa7897284fe1a3932c5d7d34a4e1d603c215a36943f3ef1fc3b2cd41375e0" exitCode=60 Oct 30 02:45:04.928571 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.928531 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" event={"ID":"4a0058da-63f6-48d6-9d96-316956c55dd4","Type":"ContainerDied","Data":"a85aa7897284fe1a3932c5d7d34a4e1d603c215a36943f3ef1fc3b2cd41375e0"} Oct 30 02:45:04.928762 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.928747 5998 scope.go:122] "RemoveContainer" containerID="a85aa7897284fe1a3932c5d7d34a4e1d603c215a36943f3ef1fc3b2cd41375e0" Oct 30 02:45:04.929079 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:04.929055 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888805-sjxb2_odh-ai-gateway-infra(4a0058da-63f6-48d6-9d96-316956c55dd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" podUID="4a0058da-63f6-48d6-9d96-316956c55dd4" Oct 30 02:45:04.930587 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.930562 5998 generic.go:356] "Generic (PLEG): container finished" podID="b9f59989-527d-411d-9fc0-ce758b48e18a" containerID="82799175b0f56c945563f8c56a785f758fcc45b4c9a0e918005b9e269ced7a99" exitCode=60 Oct 30 02:45:04.930702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.930650 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" event={"ID":"b9f59989-527d-411d-9fc0-ce758b48e18a","Type":"ContainerDied","Data":"82799175b0f56c945563f8c56a785f758fcc45b4c9a0e918005b9e269ced7a99"} Oct 30 02:45:04.930838 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.930821 5998 scope.go:122] "RemoveContainer" containerID="82799175b0f56c945563f8c56a785f758fcc45b4c9a0e918005b9e269ced7a99" Oct 30 02:45:04.931127 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:04.931075 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7_odh-ai-gateway-infra(b9f59989-527d-411d-9fc0-ce758b48e18a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" podUID="b9f59989-527d-411d-9fc0-ce758b48e18a" Oct 30 02:45:04.932591 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.932574 5998 generic.go:356] "Generic (PLEG): container finished" podID="cce212c5-9e99-4e58-8dde-89b23f8e9e2e" containerID="75e97c06d52cbf33f9ef64a178856f81ceb2c5033b03f3754c517606e0ade35c" exitCode=60 Oct 30 02:45:04.932678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.932656 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" event={"ID":"cce212c5-9e99-4e58-8dde-89b23f8e9e2e","Type":"ContainerDied","Data":"75e97c06d52cbf33f9ef64a178856f81ceb2c5033b03f3754c517606e0ade35c"} Oct 30 02:45:04.932844 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.932829 5998 scope.go:122] "RemoveContainer" containerID="75e97c06d52cbf33f9ef64a178856f81ceb2c5033b03f3754c517606e0ade35c" Oct 30 02:45:04.933160 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:04.933134 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g_odh-ai-gateway-infra(cce212c5-9e99-4e58-8dde-89b23f8e9e2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" podUID="cce212c5-9e99-4e58-8dde-89b23f8e9e2e" Oct 30 02:45:04.934555 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.934535 5998 generic.go:356] "Generic (PLEG): container finished" podID="c809da91-82e4-4c81-8644-d6409c5de91c" containerID="2b5b0dd77b0c419d53bdb8e5327097202e1f21047be211ea6e01377104b0766a" exitCode=60 Oct 30 02:45:04.934654 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.934610 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" event={"ID":"c809da91-82e4-4c81-8644-d6409c5de91c","Type":"ContainerDied","Data":"2b5b0dd77b0c419d53bdb8e5327097202e1f21047be211ea6e01377104b0766a"} Oct 30 02:45:04.934805 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.934789 5998 scope.go:122] "RemoveContainer" containerID="2b5b0dd77b0c419d53bdb8e5327097202e1f21047be211ea6e01377104b0766a" Oct 30 02:45:04.935066 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:04.935040 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld_odh-ai-gateway-infra(c809da91-82e4-4c81-8644-d6409c5de91c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" podUID="c809da91-82e4-4c81-8644-d6409c5de91c" Oct 30 02:45:04.936474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.936455 5998 generic.go:356] "Generic (PLEG): container finished" podID="5bc5e6a0-5532-497a-8244-94b6768480f2" containerID="8aaeb6f825f64a64ff0a9592aefd973b03e1b38c1347cd4d8f5f8a1edfe935d0" exitCode=60 Oct 30 02:45:04.936563 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.936518 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" event={"ID":"5bc5e6a0-5532-497a-8244-94b6768480f2","Type":"ContainerDied","Data":"8aaeb6f825f64a64ff0a9592aefd973b03e1b38c1347cd4d8f5f8a1edfe935d0"} Oct 30 02:45:04.936730 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.936713 5998 scope.go:122] "RemoveContainer" containerID="8aaeb6f825f64a64ff0a9592aefd973b03e1b38c1347cd4d8f5f8a1edfe935d0" Oct 30 02:45:04.936996 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:04.936972 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2_odh-ai-gateway-infra(5bc5e6a0-5532-497a-8244-94b6768480f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" podUID="5bc5e6a0-5532-497a-8244-94b6768480f2" Oct 30 02:45:04.938273 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.938250 5998 generic.go:356] "Generic (PLEG): container finished" podID="c560bec5-d0f1-4854-8ca0-24d44f73c750" containerID="3d117c0adf46209225d6ae72d3b8e98a942193c1a26fb7d57caf87ec81c9deb8" exitCode=60 Oct 30 02:45:04.938401 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.938327 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" event={"ID":"c560bec5-d0f1-4854-8ca0-24d44f73c750","Type":"ContainerDied","Data":"3d117c0adf46209225d6ae72d3b8e98a942193c1a26fb7d57caf87ec81c9deb8"} Oct 30 02:45:04.938532 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.938512 5998 scope.go:122] "RemoveContainer" containerID="3d117c0adf46209225d6ae72d3b8e98a942193c1a26fb7d57caf87ec81c9deb8" Oct 30 02:45:04.938763 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:04.938741 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf_odh-ai-gateway-infra(c560bec5-d0f1-4854-8ca0-24d44f73c750)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" podUID="c560bec5-d0f1-4854-8ca0-24d44f73c750" Oct 30 02:45:04.940553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.940527 5998 generic.go:356] "Generic (PLEG): container finished" podID="929aedab-934e-4cb3-b540-3dc89c8a974a" containerID="d90b664cc9ee653df11ecd82c4592cac0a415415188a0cfc126328c811c6b34b" exitCode=60 Oct 30 02:45:04.940660 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.940609 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" event={"ID":"929aedab-934e-4cb3-b540-3dc89c8a974a","Type":"ContainerDied","Data":"d90b664cc9ee653df11ecd82c4592cac0a415415188a0cfc126328c811c6b34b"} Oct 30 02:45:04.940804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.940787 5998 scope.go:122] "RemoveContainer" containerID="d90b664cc9ee653df11ecd82c4592cac0a415415188a0cfc126328c811c6b34b" Oct 30 02:45:04.941018 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:04.940997 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw_odh-ai-gateway-infra(929aedab-934e-4cb3-b540-3dc89c8a974a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" podUID="929aedab-934e-4cb3-b540-3dc89c8a974a" Oct 30 02:45:04.941584 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.941570 5998 scope.go:122] "RemoveContainer" containerID="ca42027cd31464972c0146f006c4e30489c5d9e39a8b3d08790d0a8d02a79913" Oct 30 02:45:04.942659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.942638 5998 generic.go:356] "Generic (PLEG): container finished" podID="8182c9bd-076d-4f26-8e5a-4596606d3d2d" containerID="8017f7561ba2e60dc781cc4422748b8975c147895a6b02eb595ff1a6736ef822" exitCode=60 Oct 30 02:45:04.942736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.942677 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" event={"ID":"8182c9bd-076d-4f26-8e5a-4596606d3d2d","Type":"ContainerDied","Data":"8017f7561ba2e60dc781cc4422748b8975c147895a6b02eb595ff1a6736ef822"} Oct 30 02:45:04.942893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.942875 5998 scope.go:122] "RemoveContainer" containerID="8017f7561ba2e60dc781cc4422748b8975c147895a6b02eb595ff1a6736ef822" Oct 30 02:45:04.943076 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:04.943059 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw_odh-ai-gateway-infra(8182c9bd-076d-4f26-8e5a-4596606d3d2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" podUID="8182c9bd-076d-4f26-8e5a-4596606d3d2d" Oct 30 02:45:04.956153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.956110 5998 scope.go:122] "RemoveContainer" containerID="15b07948a17216e633f36828ab2b00cf4a39931d16b4768f5bebb0b337455c6b" Oct 30 02:45:04.971526 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.971498 5998 scope.go:122] "RemoveContainer" containerID="9ddc12ef9fbe1c7275f0292dc582c1d2502eb69f5988e67321185d55e8cc1ae2" Oct 30 02:45:04.986120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:04.986062 5998 scope.go:122] "RemoveContainer" containerID="a5a1f5fc5b129c4ca8598cc363edcbbca9a5fb2594908c17dfa4bab65d4cfffc" Oct 30 02:45:05.001476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:05.001451 5998 scope.go:122] "RemoveContainer" containerID="b549850223a8e5052a5a96c2d8d9573323b469f031fe7b1ff45c14d6fb01226a" Oct 30 02:45:05.016659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:05.016632 5998 scope.go:122] "RemoveContainer" containerID="df33e81cfc287377c09d73dfb081d6547ecb81f229159fedaaa89323171b1cdc" Oct 30 02:45:05.034444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:05.034334 5998 scope.go:122] "RemoveContainer" containerID="d6830cd9274bb9e449393bfab9ef5ddef14ffd2b173418d59e0946e0fbff2815" Oct 30 02:45:05.049756 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:05.049734 5998 scope.go:122] "RemoveContainer" containerID="b624a2e6948f3a71d805ee48926a4b3aa38b59681c2e0e3ba2aff839e213ad67" Oct 30 02:45:05.065555 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:05.065532 5998 scope.go:122] "RemoveContainer" containerID="931e874b00c3a16f26e0bea073ea498eb730e7bc3980dfd539ba0268ada15120" Oct 30 02:45:05.072390 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:05.072357 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc"] Oct 30 02:45:05.075419 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:45:05.075385 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d4b059_e455_4127_bccc_da21bf6e0680.slice/crio-ae883c278cc87ca07acb14725724c071bcadead6298827c2c673f3a74d5cd79e WatchSource:0}: Error finding container ae883c278cc87ca07acb14725724c071bcadead6298827c2c673f3a74d5cd79e: Status 404 returned error can't find the container with id ae883c278cc87ca07acb14725724c071bcadead6298827c2c673f3a74d5cd79e Oct 30 02:45:05.953426 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:05.953393 5998 scope.go:122] "RemoveContainer" containerID="82799175b0f56c945563f8c56a785f758fcc45b4c9a0e918005b9e269ced7a99" Oct 30 02:45:05.953946 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:05.953755 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7_odh-ai-gateway-infra(b9f59989-527d-411d-9fc0-ce758b48e18a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" podUID="b9f59989-527d-411d-9fc0-ce758b48e18a" Oct 30 02:45:05.955322 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:05.955298 5998 scope.go:122] "RemoveContainer" containerID="75e97c06d52cbf33f9ef64a178856f81ceb2c5033b03f3754c517606e0ade35c" Oct 30 02:45:05.955596 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:05.955572 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g_odh-ai-gateway-infra(cce212c5-9e99-4e58-8dde-89b23f8e9e2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" podUID="cce212c5-9e99-4e58-8dde-89b23f8e9e2e" Oct 30 02:45:05.956765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:05.956735 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc" event={"ID":"e1d4b059-e455-4127-bccc-da21bf6e0680","Type":"ContainerStarted","Data":"97a89d83f97b9e24acfb4330a8a3669db7c78d8c11e40ec65b7be4ad51801cc5"} Oct 30 02:45:05.956918 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:05.956789 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc" event={"ID":"e1d4b059-e455-4127-bccc-da21bf6e0680","Type":"ContainerStarted","Data":"ae883c278cc87ca07acb14725724c071bcadead6298827c2c673f3a74d5cd79e"} Oct 30 02:45:05.958781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:05.958759 5998 scope.go:122] "RemoveContainer" containerID="2b5b0dd77b0c419d53bdb8e5327097202e1f21047be211ea6e01377104b0766a" Oct 30 02:45:05.959011 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:05.958991 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld_odh-ai-gateway-infra(c809da91-82e4-4c81-8644-d6409c5de91c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" podUID="c809da91-82e4-4c81-8644-d6409c5de91c" Oct 30 02:45:05.960404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:05.960384 5998 scope.go:122] "RemoveContainer" containerID="8aaeb6f825f64a64ff0a9592aefd973b03e1b38c1347cd4d8f5f8a1edfe935d0" Oct 30 02:45:05.960607 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:05.960587 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2_odh-ai-gateway-infra(5bc5e6a0-5532-497a-8244-94b6768480f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" podUID="5bc5e6a0-5532-497a-8244-94b6768480f2" Oct 30 02:45:05.961921 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:05.961903 5998 scope.go:122] "RemoveContainer" containerID="3d117c0adf46209225d6ae72d3b8e98a942193c1a26fb7d57caf87ec81c9deb8" Oct 30 02:45:05.962165 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:05.962142 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf_odh-ai-gateway-infra(c560bec5-d0f1-4854-8ca0-24d44f73c750)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" podUID="c560bec5-d0f1-4854-8ca0-24d44f73c750" Oct 30 02:45:05.963451 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:05.963431 5998 scope.go:122] "RemoveContainer" containerID="d90b664cc9ee653df11ecd82c4592cac0a415415188a0cfc126328c811c6b34b" Oct 30 02:45:05.963666 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:05.963645 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw_odh-ai-gateway-infra(929aedab-934e-4cb3-b540-3dc89c8a974a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" podUID="929aedab-934e-4cb3-b540-3dc89c8a974a" Oct 30 02:45:05.964986 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:05.964967 5998 scope.go:122] "RemoveContainer" containerID="8017f7561ba2e60dc781cc4422748b8975c147895a6b02eb595ff1a6736ef822" Oct 30 02:45:05.965181 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:05.965162 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw_odh-ai-gateway-infra(8182c9bd-076d-4f26-8e5a-4596606d3d2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" podUID="8182c9bd-076d-4f26-8e5a-4596606d3d2d" Oct 30 02:45:05.966440 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:05.966423 5998 scope.go:122] "RemoveContainer" containerID="a4a0fb458be0d3a6b59f584c9f0d7290502b0d64ba770b4b8d3286b0b5d3f7eb" Oct 30 02:45:05.966659 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:05.966636 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf_odh-ai-gateway-infra(e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" podUID="e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4" Oct 30 02:45:05.968077 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:05.968057 5998 scope.go:122] "RemoveContainer" containerID="6b7d910316d04683370686f4148be9d21d5f1b679e74bb79dc9f2fc295a5c7ed" Oct 30 02:45:05.968346 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:05.968310 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v_odh-ai-gateway-infra(662d5231-266e-4ff9-8e5a-4409da2c7d82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" podUID="662d5231-266e-4ff9-8e5a-4409da2c7d82" Oct 30 02:45:05.969761 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:05.969744 5998 scope.go:122] "RemoveContainer" containerID="a85aa7897284fe1a3932c5d7d34a4e1d603c215a36943f3ef1fc3b2cd41375e0" Oct 30 02:45:05.969926 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:05.969910 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888805-sjxb2_odh-ai-gateway-infra(4a0058da-63f6-48d6-9d96-316956c55dd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" podUID="4a0058da-63f6-48d6-9d96-316956c55dd4" Oct 30 02:45:06.476167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:06.476128 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:06.488890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:06.488827 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc" podStartSLOduration=2.488812688 podStartE2EDuration="2.488812688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:06.418715918 +0000 UTC m=+4535.997507001" watchObservedRunningTime="2026-10-30 02:45:06.488812688 +0000 UTC m=+4536.067603769" Oct 30 02:45:06.980745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:06.980700 5998 generic.go:356] "Generic (PLEG): container finished" podID="e1d4b059-e455-4127-bccc-da21bf6e0680" containerID="97a89d83f97b9e24acfb4330a8a3669db7c78d8c11e40ec65b7be4ad51801cc5" exitCode=0 Oct 30 02:45:06.981301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:06.980786 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc" event={"ID":"e1d4b059-e455-4127-bccc-da21bf6e0680","Type":"ContainerDied","Data":"97a89d83f97b9e24acfb4330a8a3669db7c78d8c11e40ec65b7be4ad51801cc5"} Oct 30 02:45:08.180784 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:08.180751 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc" Oct 30 02:45:08.259464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:08.259374 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1d4b059-e455-4127-bccc-da21bf6e0680-kube-api-access-twjgj\" (UniqueName: \"kubernetes.io/projected/e1d4b059-e455-4127-bccc-da21bf6e0680-kube-api-access-twjgj\") pod \"e1d4b059-e455-4127-bccc-da21bf6e0680\" (UID: \"e1d4b059-e455-4127-bccc-da21bf6e0680\") " Oct 30 02:45:08.259635 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:08.259480 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1d4b059-e455-4127-bccc-da21bf6e0680-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d4b059-e455-4127-bccc-da21bf6e0680-maas-api-xks-ca\") pod \"e1d4b059-e455-4127-bccc-da21bf6e0680\" (UID: \"e1d4b059-e455-4127-bccc-da21bf6e0680\") " Oct 30 02:45:08.259635 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:08.259522 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1d4b059-e455-4127-bccc-da21bf6e0680-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d4b059-e455-4127-bccc-da21bf6e0680-maas-api-openshift-ca\") pod \"e1d4b059-e455-4127-bccc-da21bf6e0680\" (UID: \"e1d4b059-e455-4127-bccc-da21bf6e0680\") " Oct 30 02:45:08.259962 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:08.259922 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1d4b059-e455-4127-bccc-da21bf6e0680-maas-api-openshift-ca" pod "e1d4b059-e455-4127-bccc-da21bf6e0680" (UID: "e1d4b059-e455-4127-bccc-da21bf6e0680"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:08.260225 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:08.260196 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d4b059-e455-4127-bccc-da21bf6e0680-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:08.262450 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:08.262418 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d4b059-e455-4127-bccc-da21bf6e0680-kube-api-access-twjgj" pod "e1d4b059-e455-4127-bccc-da21bf6e0680" (UID: "e1d4b059-e455-4127-bccc-da21bf6e0680"). InnerVolumeSpecName "kube-api-access-twjgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:08.262559 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:08.262526 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1d4b059-e455-4127-bccc-da21bf6e0680-maas-api-xks-ca" pod "e1d4b059-e455-4127-bccc-da21bf6e0680" (UID: "e1d4b059-e455-4127-bccc-da21bf6e0680"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:45:08.361898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:08.361850 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twjgj\" (UniqueName: \"kubernetes.io/projected/e1d4b059-e455-4127-bccc-da21bf6e0680-kube-api-access-twjgj\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:08.361898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:08.361900 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d4b059-e455-4127-bccc-da21bf6e0680-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:08.378956 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:08.378920 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:08.996455 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:08.996419 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc" Oct 30 02:45:08.996671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:08.996454 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc" event={"ID":"e1d4b059-e455-4127-bccc-da21bf6e0680","Type":"ContainerDied","Data":"ae883c278cc87ca07acb14725724c071bcadead6298827c2c673f3a74d5cd79e"} Oct 30 02:45:08.996671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:08.996502 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae883c278cc87ca07acb14725724c071bcadead6298827c2c673f3a74d5cd79e" Oct 30 02:45:09.887122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:09.887058 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:11.457065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:11.457016 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:13.023060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:13.023021 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:14.491393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:14.491346 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:14.535242 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:14.535205 5998 scope.go:122] "RemoveContainer" containerID="8aaeb6f825f64a64ff0a9592aefd973b03e1b38c1347cd4d8f5f8a1edfe935d0" Oct 30 02:45:15.075746 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:15.075661 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" event={"ID":"5bc5e6a0-5532-497a-8244-94b6768480f2","Type":"ContainerStarted","Data":"4eb51c2309ac8161c7b6d6e2c191954658beb613d2f9559887c789a7d1694b02"} Oct 30 02:45:15.295402 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:15.295364 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:15.532304 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:15.532268 5998 scope.go:122] "RemoveContainer" containerID="d90b664cc9ee653df11ecd82c4592cac0a415415188a0cfc126328c811c6b34b" Oct 30 02:45:15.532798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:15.532395 5998 scope.go:122] "RemoveContainer" containerID="8017f7561ba2e60dc781cc4422748b8975c147895a6b02eb595ff1a6736ef822" Oct 30 02:45:16.086502 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:16.086414 5998 generic.go:356] "Generic (PLEG): container finished" podID="5bc5e6a0-5532-497a-8244-94b6768480f2" containerID="4eb51c2309ac8161c7b6d6e2c191954658beb613d2f9559887c789a7d1694b02" exitCode=60 Oct 30 02:45:16.086502 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:16.086484 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" event={"ID":"5bc5e6a0-5532-497a-8244-94b6768480f2","Type":"ContainerDied","Data":"4eb51c2309ac8161c7b6d6e2c191954658beb613d2f9559887c789a7d1694b02"} Oct 30 02:45:16.086722 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:16.086550 5998 scope.go:122] "RemoveContainer" containerID="8aaeb6f825f64a64ff0a9592aefd973b03e1b38c1347cd4d8f5f8a1edfe935d0" Oct 30 02:45:16.087053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:16.087029 5998 scope.go:122] "RemoveContainer" containerID="4eb51c2309ac8161c7b6d6e2c191954658beb613d2f9559887c789a7d1694b02" Oct 30 02:45:16.087462 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:16.087429 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2_odh-ai-gateway-infra(5bc5e6a0-5532-497a-8244-94b6768480f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" podUID="5bc5e6a0-5532-497a-8244-94b6768480f2" Oct 30 02:45:16.089421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:16.089399 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" event={"ID":"929aedab-934e-4cb3-b540-3dc89c8a974a","Type":"ContainerStarted","Data":"778e6b9df29e4ceb84d8040961d00080bb5d40a1394348009158a790107c7550"} Oct 30 02:45:16.091466 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:16.091444 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" event={"ID":"8182c9bd-076d-4f26-8e5a-4596606d3d2d","Type":"ContainerStarted","Data":"0df8960886013df335b151e977f528a7768e9dfda22696010f955b82a386d8da"} Oct 30 02:45:16.133826 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:16.133785 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2"] Oct 30 02:45:16.188155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:16.186259 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:16.532194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:16.532165 5998 scope.go:122] "RemoveContainer" containerID="2b5b0dd77b0c419d53bdb8e5327097202e1f21047be211ea6e01377104b0766a" Oct 30 02:45:17.102172 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.102056 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" event={"ID":"c809da91-82e4-4c81-8644-d6409c5de91c","Type":"ContainerStarted","Data":"a7dda9b140cdc81f84e40f0a55fc99bf7106064f6c5b35baf6939f7a35412e76"} Oct 30 02:45:17.106068 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.106039 5998 generic.go:356] "Generic (PLEG): container finished" podID="929aedab-934e-4cb3-b540-3dc89c8a974a" containerID="778e6b9df29e4ceb84d8040961d00080bb5d40a1394348009158a790107c7550" exitCode=60 Oct 30 02:45:17.106220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.106122 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" event={"ID":"929aedab-934e-4cb3-b540-3dc89c8a974a","Type":"ContainerDied","Data":"778e6b9df29e4ceb84d8040961d00080bb5d40a1394348009158a790107c7550"} Oct 30 02:45:17.106220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.106196 5998 scope.go:122] "RemoveContainer" containerID="d90b664cc9ee653df11ecd82c4592cac0a415415188a0cfc126328c811c6b34b" Oct 30 02:45:17.106600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.106580 5998 scope.go:122] "RemoveContainer" containerID="778e6b9df29e4ceb84d8040961d00080bb5d40a1394348009158a790107c7550" Oct 30 02:45:17.106967 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:17.106940 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw_odh-ai-gateway-infra(929aedab-934e-4cb3-b540-3dc89c8a974a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" podUID="929aedab-934e-4cb3-b540-3dc89c8a974a" Oct 30 02:45:17.108421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.108397 5998 generic.go:356] "Generic (PLEG): container finished" podID="8182c9bd-076d-4f26-8e5a-4596606d3d2d" containerID="0df8960886013df335b151e977f528a7768e9dfda22696010f955b82a386d8da" exitCode=60 Oct 30 02:45:17.108523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.108437 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" event={"ID":"8182c9bd-076d-4f26-8e5a-4596606d3d2d","Type":"ContainerDied","Data":"0df8960886013df335b151e977f528a7768e9dfda22696010f955b82a386d8da"} Oct 30 02:45:17.108845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.108829 5998 scope.go:122] "RemoveContainer" containerID="0df8960886013df335b151e977f528a7768e9dfda22696010f955b82a386d8da" Oct 30 02:45:17.109161 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:17.109138 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw_odh-ai-gateway-infra(8182c9bd-076d-4f26-8e5a-4596606d3d2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" podUID="8182c9bd-076d-4f26-8e5a-4596606d3d2d" Oct 30 02:45:17.121992 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.121969 5998 scope.go:122] "RemoveContainer" containerID="8017f7561ba2e60dc781cc4422748b8975c147895a6b02eb595ff1a6736ef822" Oct 30 02:45:17.237338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.237301 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw"] Oct 30 02:45:17.250230 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.250197 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw"] Oct 30 02:45:17.278557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.278525 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" Oct 30 02:45:17.390219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.390123 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bc5e6a0-5532-497a-8244-94b6768480f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc5e6a0-5532-497a-8244-94b6768480f2-maas-api-service-ca\") pod \"5bc5e6a0-5532-497a-8244-94b6768480f2\" (UID: \"5bc5e6a0-5532-497a-8244-94b6768480f2\") " Oct 30 02:45:17.390416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.390381 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc5e6a0-5532-497a-8244-94b6768480f2-kube-api-access-tm6hl\" (UniqueName: \"kubernetes.io/projected/5bc5e6a0-5532-497a-8244-94b6768480f2-kube-api-access-tm6hl\") pod \"5bc5e6a0-5532-497a-8244-94b6768480f2\" (UID: \"5bc5e6a0-5532-497a-8244-94b6768480f2\") " Oct 30 02:45:17.390670 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.390643 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc5e6a0-5532-497a-8244-94b6768480f2-maas-api-service-ca" pod "5bc5e6a0-5532-497a-8244-94b6768480f2" (UID: "5bc5e6a0-5532-497a-8244-94b6768480f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:17.390890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.390873 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bc5e6a0-5532-497a-8244-94b6768480f2-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:17.392573 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.392542 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc5e6a0-5532-497a-8244-94b6768480f2-kube-api-access-tm6hl" pod "5bc5e6a0-5532-497a-8244-94b6768480f2" (UID: "5bc5e6a0-5532-497a-8244-94b6768480f2"). InnerVolumeSpecName "kube-api-access-tm6hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:17.492293 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.492258 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm6hl\" (UniqueName: \"kubernetes.io/projected/5bc5e6a0-5532-497a-8244-94b6768480f2-kube-api-access-tm6hl\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:17.532373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.532335 5998 scope.go:122] "RemoveContainer" containerID="82799175b0f56c945563f8c56a785f758fcc45b4c9a0e918005b9e269ced7a99" Oct 30 02:45:17.532559 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:17.532407 5998 scope.go:122] "RemoveContainer" containerID="6b7d910316d04683370686f4148be9d21d5f1b679e74bb79dc9f2fc295a5c7ed" Oct 30 02:45:18.041874 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.041838 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:18.129585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.129549 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" event={"ID":"662d5231-266e-4ff9-8e5a-4409da2c7d82","Type":"ContainerStarted","Data":"30e0fbce5e56cbadac5fc9c71c848e98652564457d2396b7db4a13dff0a201cc"} Oct 30 02:45:18.132459 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.132344 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" event={"ID":"b9f59989-527d-411d-9fc0-ce758b48e18a","Type":"ContainerStarted","Data":"6a3e4e488f28cc938c9dfbf552dbbb657fd198b8f40d44ce54f9d9b0bf809d62"} Oct 30 02:45:18.134943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.134916 5998 generic.go:356] "Generic (PLEG): container finished" podID="c809da91-82e4-4c81-8644-d6409c5de91c" containerID="a7dda9b140cdc81f84e40f0a55fc99bf7106064f6c5b35baf6939f7a35412e76" exitCode=60 Oct 30 02:45:18.135076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.134979 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" event={"ID":"c809da91-82e4-4c81-8644-d6409c5de91c","Type":"ContainerDied","Data":"a7dda9b140cdc81f84e40f0a55fc99bf7106064f6c5b35baf6939f7a35412e76"} Oct 30 02:45:18.135076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.135059 5998 scope.go:122] "RemoveContainer" containerID="2b5b0dd77b0c419d53bdb8e5327097202e1f21047be211ea6e01377104b0766a" Oct 30 02:45:18.135477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.135453 5998 scope.go:122] "RemoveContainer" containerID="a7dda9b140cdc81f84e40f0a55fc99bf7106064f6c5b35baf6939f7a35412e76" Oct 30 02:45:18.135910 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:18.135821 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld_odh-ai-gateway-infra(c809da91-82e4-4c81-8644-d6409c5de91c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" podUID="c809da91-82e4-4c81-8644-d6409c5de91c" Oct 30 02:45:18.138414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.138369 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" event={"ID":"5bc5e6a0-5532-497a-8244-94b6768480f2","Type":"ContainerDied","Data":"a403dd30739d9888039678ecbff33f9da936aefc6a5e237700b97fb76aaab654"} Oct 30 02:45:18.138609 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.138428 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2" Oct 30 02:45:18.179500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.176272 5998 scope.go:122] "RemoveContainer" containerID="4eb51c2309ac8161c7b6d6e2c191954658beb613d2f9559887c789a7d1694b02" Oct 30 02:45:18.185253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.185223 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld"] Oct 30 02:45:18.246539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.246476 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2"] Oct 30 02:45:18.250069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.250042 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-a0fc73-29888805-7sgn2"] Oct 30 02:45:18.310104 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.310057 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" Oct 30 02:45:18.344768 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.344743 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" Oct 30 02:45:18.407918 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.407885 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/929aedab-934e-4cb3-b540-3dc89c8a974a-kube-api-access-4xfs9\" (UniqueName: \"kubernetes.io/projected/929aedab-934e-4cb3-b540-3dc89c8a974a-kube-api-access-4xfs9\") pod \"929aedab-934e-4cb3-b540-3dc89c8a974a\" (UID: \"929aedab-934e-4cb3-b540-3dc89c8a974a\") " Oct 30 02:45:18.408267 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.408240 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/929aedab-934e-4cb3-b540-3dc89c8a974a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/929aedab-934e-4cb3-b540-3dc89c8a974a-maas-api-service-ca\") pod \"929aedab-934e-4cb3-b540-3dc89c8a974a\" (UID: \"929aedab-934e-4cb3-b540-3dc89c8a974a\") " Oct 30 02:45:18.408571 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.408547 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929aedab-934e-4cb3-b540-3dc89c8a974a-maas-api-service-ca" pod "929aedab-934e-4cb3-b540-3dc89c8a974a" (UID: "929aedab-934e-4cb3-b540-3dc89c8a974a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:18.408764 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.408743 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/929aedab-934e-4cb3-b540-3dc89c8a974a-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:18.410810 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.410779 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929aedab-934e-4cb3-b540-3dc89c8a974a-kube-api-access-4xfs9" pod "929aedab-934e-4cb3-b540-3dc89c8a974a" (UID: "929aedab-934e-4cb3-b540-3dc89c8a974a"). InnerVolumeSpecName "kube-api-access-4xfs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:18.435155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.435114 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh"] Oct 30 02:45:18.436049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436015 5998 cpu_manager.go:414] "containerMap: removing container" podUID="5bc5e6a0-5532-497a-8244-94b6768480f2" containerName="cleanup" Oct 30 02:45:18.436049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436044 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc5e6a0-5532-497a-8244-94b6768480f2" containerName="cleanup" Oct 30 02:45:18.436297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436059 5998 cpu_manager.go:414] "containerMap: removing container" podUID="5bc5e6a0-5532-497a-8244-94b6768480f2" containerName="cleanup" Oct 30 02:45:18.436297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436067 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc5e6a0-5532-497a-8244-94b6768480f2" containerName="cleanup" Oct 30 02:45:18.436297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436112 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8182c9bd-076d-4f26-8e5a-4596606d3d2d" containerName="cleanup" Oct 30 02:45:18.436297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436120 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8182c9bd-076d-4f26-8e5a-4596606d3d2d" containerName="cleanup" Oct 30 02:45:18.436297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436134 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8182c9bd-076d-4f26-8e5a-4596606d3d2d" containerName="cleanup" Oct 30 02:45:18.436297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436142 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8182c9bd-076d-4f26-8e5a-4596606d3d2d" containerName="cleanup" Oct 30 02:45:18.436297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436152 5998 cpu_manager.go:414] "containerMap: removing container" podUID="929aedab-934e-4cb3-b540-3dc89c8a974a" containerName="cleanup" Oct 30 02:45:18.436297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436159 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929aedab-934e-4cb3-b540-3dc89c8a974a" containerName="cleanup" Oct 30 02:45:18.436297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436187 5998 cpu_manager.go:414] "containerMap: removing container" podUID="929aedab-934e-4cb3-b540-3dc89c8a974a" containerName="cleanup" Oct 30 02:45:18.436297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436194 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929aedab-934e-4cb3-b540-3dc89c8a974a" containerName="cleanup" Oct 30 02:45:18.436297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436209 5998 cpu_manager.go:414] "containerMap: removing container" podUID="929aedab-934e-4cb3-b540-3dc89c8a974a" containerName="cleanup" Oct 30 02:45:18.436297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436216 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="929aedab-934e-4cb3-b540-3dc89c8a974a" containerName="cleanup" Oct 30 02:45:18.436297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436232 5998 cpu_manager.go:414] "containerMap: removing container" podUID="e1d4b059-e455-4127-bccc-da21bf6e0680" containerName="revoke-keys" Oct 30 02:45:18.436297 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436239 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d4b059-e455-4127-bccc-da21bf6e0680" containerName="revoke-keys" Oct 30 02:45:18.437018 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436433 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8182c9bd-076d-4f26-8e5a-4596606d3d2d" containerName="cleanup" Oct 30 02:45:18.437018 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436451 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8182c9bd-076d-4f26-8e5a-4596606d3d2d" containerName="cleanup" Oct 30 02:45:18.437018 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436460 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="929aedab-934e-4cb3-b540-3dc89c8a974a" containerName="cleanup" Oct 30 02:45:18.437018 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436471 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc5e6a0-5532-497a-8244-94b6768480f2" containerName="cleanup" Oct 30 02:45:18.437018 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436480 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="929aedab-934e-4cb3-b540-3dc89c8a974a" containerName="cleanup" Oct 30 02:45:18.437018 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436489 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="929aedab-934e-4cb3-b540-3dc89c8a974a" containerName="cleanup" Oct 30 02:45:18.437018 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436501 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d4b059-e455-4127-bccc-da21bf6e0680" containerName="revoke-keys" Oct 30 02:45:18.437018 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.436519 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc5e6a0-5532-497a-8244-94b6768480f2" containerName="cleanup" Oct 30 02:45:18.444850 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.444813 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh" Oct 30 02:45:18.459505 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.459458 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:45:18.463212 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.463180 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh"] Oct 30 02:45:18.509513 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.509483 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8182c9bd-076d-4f26-8e5a-4596606d3d2d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8182c9bd-076d-4f26-8e5a-4596606d3d2d-maas-api-service-ca\") pod \"8182c9bd-076d-4f26-8e5a-4596606d3d2d\" (UID: \"8182c9bd-076d-4f26-8e5a-4596606d3d2d\") " Oct 30 02:45:18.509740 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.509722 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8182c9bd-076d-4f26-8e5a-4596606d3d2d-kube-api-access-rrlqs\" (UniqueName: \"kubernetes.io/projected/8182c9bd-076d-4f26-8e5a-4596606d3d2d-kube-api-access-rrlqs\") pod \"8182c9bd-076d-4f26-8e5a-4596606d3d2d\" (UID: \"8182c9bd-076d-4f26-8e5a-4596606d3d2d\") " Oct 30 02:45:18.509906 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.509881 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8182c9bd-076d-4f26-8e5a-4596606d3d2d-maas-api-service-ca" pod "8182c9bd-076d-4f26-8e5a-4596606d3d2d" (UID: "8182c9bd-076d-4f26-8e5a-4596606d3d2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:18.510217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.510193 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8182c9bd-076d-4f26-8e5a-4596606d3d2d-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:18.510217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.510214 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xfs9\" (UniqueName: \"kubernetes.io/projected/929aedab-934e-4cb3-b540-3dc89c8a974a-kube-api-access-4xfs9\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:18.511851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.511823 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8182c9bd-076d-4f26-8e5a-4596606d3d2d-kube-api-access-rrlqs" pod "8182c9bd-076d-4f26-8e5a-4596606d3d2d" (UID: "8182c9bd-076d-4f26-8e5a-4596606d3d2d"). InnerVolumeSpecName "kube-api-access-rrlqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:18.532707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.532679 5998 scope.go:122] "RemoveContainer" containerID="3d117c0adf46209225d6ae72d3b8e98a942193c1a26fb7d57caf87ec81c9deb8" Oct 30 02:45:18.532875 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.532764 5998 scope.go:122] "RemoveContainer" containerID="a4a0fb458be0d3a6b59f584c9f0d7290502b0d64ba770b4b8d3286b0b5d3f7eb" Oct 30 02:45:18.533044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.533023 5998 scope.go:122] "RemoveContainer" containerID="75e97c06d52cbf33f9ef64a178856f81ceb2c5033b03f3754c517606e0ade35c" Oct 30 02:45:18.533447 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.533425 5998 scope.go:122] "RemoveContainer" containerID="a85aa7897284fe1a3932c5d7d34a4e1d603c215a36943f3ef1fc3b2cd41375e0" Oct 30 02:45:18.542742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.542720 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bc5e6a0-5532-497a-8244-94b6768480f2" path="/var/lib/kubelet/pods/5bc5e6a0-5532-497a-8244-94b6768480f2/volumes" Oct 30 02:45:18.611691 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.611657 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ca672a-fa36-4182-8cac-59d4d8b992b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdbdb098-jlbwh\" (UID: \"c5ca672a-fa36-4182-8cac-59d4d8b992b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh" Oct 30 02:45:18.611840 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.611708 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8nvt\" (UniqueName: \"kubernetes.io/projected/c5ca672a-fa36-4182-8cac-59d4d8b992b6-kube-api-access-r8nvt\") pod \"maas-api-revoke-sub-cdbdb098-jlbwh\" (UID: \"c5ca672a-fa36-4182-8cac-59d4d8b992b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh" Oct 30 02:45:18.611938 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.611921 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ca672a-fa36-4182-8cac-59d4d8b992b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdbdb098-jlbwh\" (UID: \"c5ca672a-fa36-4182-8cac-59d4d8b992b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh" Oct 30 02:45:18.612122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.612108 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrlqs\" (UniqueName: \"kubernetes.io/projected/8182c9bd-076d-4f26-8e5a-4596606d3d2d-kube-api-access-rrlqs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:18.713645 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.713606 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ca672a-fa36-4182-8cac-59d4d8b992b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdbdb098-jlbwh\" (UID: \"c5ca672a-fa36-4182-8cac-59d4d8b992b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh" Oct 30 02:45:18.713832 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.713666 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8nvt\" (UniqueName: \"kubernetes.io/projected/c5ca672a-fa36-4182-8cac-59d4d8b992b6-kube-api-access-r8nvt\") pod \"maas-api-revoke-sub-cdbdb098-jlbwh\" (UID: \"c5ca672a-fa36-4182-8cac-59d4d8b992b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh" Oct 30 02:45:18.713898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.713856 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ca672a-fa36-4182-8cac-59d4d8b992b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdbdb098-jlbwh\" (UID: \"c5ca672a-fa36-4182-8cac-59d4d8b992b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh" Oct 30 02:45:18.714789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.714761 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ca672a-fa36-4182-8cac-59d4d8b992b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdbdb098-jlbwh\" (UID: \"c5ca672a-fa36-4182-8cac-59d4d8b992b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh" Oct 30 02:45:18.803154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.803060 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ca672a-fa36-4182-8cac-59d4d8b992b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdbdb098-jlbwh\" (UID: \"c5ca672a-fa36-4182-8cac-59d4d8b992b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh" Oct 30 02:45:18.803632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:18.803599 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8nvt\" (UniqueName: \"kubernetes.io/projected/c5ca672a-fa36-4182-8cac-59d4d8b992b6-kube-api-access-r8nvt\") pod \"maas-api-revoke-sub-cdbdb098-jlbwh\" (UID: \"c5ca672a-fa36-4182-8cac-59d4d8b992b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh" Oct 30 02:45:19.073743 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.073658 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh" Oct 30 02:45:19.171114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.171057 5998 generic.go:356] "Generic (PLEG): container finished" podID="b9f59989-527d-411d-9fc0-ce758b48e18a" containerID="6a3e4e488f28cc938c9dfbf552dbbb657fd198b8f40d44ce54f9d9b0bf809d62" exitCode=60 Oct 30 02:45:19.171582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.171200 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" event={"ID":"b9f59989-527d-411d-9fc0-ce758b48e18a","Type":"ContainerDied","Data":"6a3e4e488f28cc938c9dfbf552dbbb657fd198b8f40d44ce54f9d9b0bf809d62"} Oct 30 02:45:19.171582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.171260 5998 scope.go:122] "RemoveContainer" containerID="82799175b0f56c945563f8c56a785f758fcc45b4c9a0e918005b9e269ced7a99" Oct 30 02:45:19.171770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.171747 5998 scope.go:122] "RemoveContainer" containerID="6a3e4e488f28cc938c9dfbf552dbbb657fd198b8f40d44ce54f9d9b0bf809d62" Oct 30 02:45:19.172186 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:19.172156 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7_odh-ai-gateway-infra(b9f59989-527d-411d-9fc0-ce758b48e18a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" podUID="b9f59989-527d-411d-9fc0-ce758b48e18a" Oct 30 02:45:19.173726 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.173703 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" event={"ID":"cce212c5-9e99-4e58-8dde-89b23f8e9e2e","Type":"ContainerStarted","Data":"8f30423f1a90e51527f5adfe78bd0619b81e4c78d6ea1cdecd278ed94de7a13e"} Oct 30 02:45:19.180855 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.179874 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" event={"ID":"c560bec5-d0f1-4854-8ca0-24d44f73c750","Type":"ContainerStarted","Data":"7819c388f08e3745e3ba8a96ff0fac95f3a4ffcc1733f1a34164ce968ac68d09"} Oct 30 02:45:19.182661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.182069 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" event={"ID":"929aedab-934e-4cb3-b540-3dc89c8a974a","Type":"ContainerDied","Data":"1b409615a46652a41e8587066dbac4944826aab4415815fa5ce85c9a97435219"} Oct 30 02:45:19.182661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.182249 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw" Oct 30 02:45:19.185143 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.184019 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" event={"ID":"8182c9bd-076d-4f26-8e5a-4596606d3d2d","Type":"ContainerDied","Data":"a35983d4a607b39529bf6e1144d808a970c01655d22b00d67b17a5eda7a3715b"} Oct 30 02:45:19.185143 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.184113 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw" Oct 30 02:45:19.193023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.192441 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" event={"ID":"e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4","Type":"ContainerStarted","Data":"a692cabf029e9dc783ec4c5a60c1012e18544e01cde29ab0d76e46b6978454e2"} Oct 30 02:45:19.195474 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.195441 5998 generic.go:356] "Generic (PLEG): container finished" podID="662d5231-266e-4ff9-8e5a-4409da2c7d82" containerID="30e0fbce5e56cbadac5fc9c71c848e98652564457d2396b7db4a13dff0a201cc" exitCode=60 Oct 30 02:45:19.195567 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.195535 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" event={"ID":"662d5231-266e-4ff9-8e5a-4409da2c7d82","Type":"ContainerDied","Data":"30e0fbce5e56cbadac5fc9c71c848e98652564457d2396b7db4a13dff0a201cc"} Oct 30 02:45:19.195932 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.195912 5998 scope.go:122] "RemoveContainer" containerID="30e0fbce5e56cbadac5fc9c71c848e98652564457d2396b7db4a13dff0a201cc" Oct 30 02:45:19.196260 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:19.196235 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v_odh-ai-gateway-infra(662d5231-266e-4ff9-8e5a-4409da2c7d82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" podUID="662d5231-266e-4ff9-8e5a-4409da2c7d82" Oct 30 02:45:19.196968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.196951 5998 scope.go:122] "RemoveContainer" containerID="778e6b9df29e4ceb84d8040961d00080bb5d40a1394348009158a790107c7550" Oct 30 02:45:19.199638 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.199615 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" event={"ID":"4a0058da-63f6-48d6-9d96-316956c55dd4","Type":"ContainerStarted","Data":"d8ba0ae6afdb5341163f0651c5cd7891e1478efe4332899ec3762a3eb6106f0e"} Oct 30 02:45:19.217034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.217005 5998 scope.go:122] "RemoveContainer" containerID="0df8960886013df335b151e977f528a7768e9dfda22696010f955b82a386d8da" Oct 30 02:45:19.230545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.230512 5998 scope.go:122] "RemoveContainer" containerID="6b7d910316d04683370686f4148be9d21d5f1b679e74bb79dc9f2fc295a5c7ed" Oct 30 02:45:19.295965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.295928 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7"] Oct 30 02:45:19.334467 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.334365 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v"] Oct 30 02:45:19.363126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.353283 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" Oct 30 02:45:19.363126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.359483 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw"] Oct 30 02:45:19.363126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.362402 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-78d5ff-29888805-hksvw"] Oct 30 02:45:19.386588 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.386564 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh"] Oct 30 02:45:19.389610 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:45:19.389582 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ca672a_fa36_4182_8cac_59d4d8b992b6.slice/crio-6c034cce34b20fa0b89f8cba96c9a226f80ffd605b27bf0e47b6aa9a6823449d WatchSource:0}: Error finding container 6c034cce34b20fa0b89f8cba96c9a226f80ffd605b27bf0e47b6aa9a6823449d: Status 404 returned error can't find the container with id 6c034cce34b20fa0b89f8cba96c9a226f80ffd605b27bf0e47b6aa9a6823449d Oct 30 02:45:19.468008 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.467969 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw"] Oct 30 02:45:19.478537 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.478505 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6b8bfa-29888805-lxhkw"] Oct 30 02:45:19.525646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.525612 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c809da91-82e4-4c81-8644-d6409c5de91c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c809da91-82e4-4c81-8644-d6409c5de91c-maas-api-service-ca\") pod \"c809da91-82e4-4c81-8644-d6409c5de91c\" (UID: \"c809da91-82e4-4c81-8644-d6409c5de91c\") " Oct 30 02:45:19.525820 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.525767 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c809da91-82e4-4c81-8644-d6409c5de91c-kube-api-access-8gd2p\" (UniqueName: \"kubernetes.io/projected/c809da91-82e4-4c81-8644-d6409c5de91c-kube-api-access-8gd2p\") pod \"c809da91-82e4-4c81-8644-d6409c5de91c\" (UID: \"c809da91-82e4-4c81-8644-d6409c5de91c\") " Oct 30 02:45:19.526032 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.526005 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c809da91-82e4-4c81-8644-d6409c5de91c-maas-api-service-ca" pod "c809da91-82e4-4c81-8644-d6409c5de91c" (UID: "c809da91-82e4-4c81-8644-d6409c5de91c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:19.526289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.526270 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c809da91-82e4-4c81-8644-d6409c5de91c-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:19.527943 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.527917 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c809da91-82e4-4c81-8644-d6409c5de91c-kube-api-access-8gd2p" pod "c809da91-82e4-4c81-8644-d6409c5de91c" (UID: "c809da91-82e4-4c81-8644-d6409c5de91c"). InnerVolumeSpecName "kube-api-access-8gd2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:19.627736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.627704 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gd2p\" (UniqueName: \"kubernetes.io/projected/c809da91-82e4-4c81-8644-d6409c5de91c-kube-api-access-8gd2p\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:19.734313 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:19.734275 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:20.211023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.210984 5998 generic.go:356] "Generic (PLEG): container finished" podID="e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4" containerID="a692cabf029e9dc783ec4c5a60c1012e18544e01cde29ab0d76e46b6978454e2" exitCode=60 Oct 30 02:45:20.211539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.211069 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" event={"ID":"e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4","Type":"ContainerDied","Data":"a692cabf029e9dc783ec4c5a60c1012e18544e01cde29ab0d76e46b6978454e2"} Oct 30 02:45:20.211539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.211163 5998 scope.go:122] "RemoveContainer" containerID="a4a0fb458be0d3a6b59f584c9f0d7290502b0d64ba770b4b8d3286b0b5d3f7eb" Oct 30 02:45:20.211689 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.211561 5998 scope.go:122] "RemoveContainer" containerID="a692cabf029e9dc783ec4c5a60c1012e18544e01cde29ab0d76e46b6978454e2" Oct 30 02:45:20.211939 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:20.211911 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf_odh-ai-gateway-infra(e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" podUID="e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4" Oct 30 02:45:20.215969 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.215946 5998 generic.go:356] "Generic (PLEG): container finished" podID="4a0058da-63f6-48d6-9d96-316956c55dd4" containerID="d8ba0ae6afdb5341163f0651c5cd7891e1478efe4332899ec3762a3eb6106f0e" exitCode=60 Oct 30 02:45:20.216111 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.216061 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" event={"ID":"4a0058da-63f6-48d6-9d96-316956c55dd4","Type":"ContainerDied","Data":"d8ba0ae6afdb5341163f0651c5cd7891e1478efe4332899ec3762a3eb6106f0e"} Oct 30 02:45:20.216522 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.216499 5998 scope.go:122] "RemoveContainer" containerID="d8ba0ae6afdb5341163f0651c5cd7891e1478efe4332899ec3762a3eb6106f0e" Oct 30 02:45:20.216874 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:20.216847 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888805-sjxb2_odh-ai-gateway-infra(4a0058da-63f6-48d6-9d96-316956c55dd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" podUID="4a0058da-63f6-48d6-9d96-316956c55dd4" Oct 30 02:45:20.222634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.222605 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh" event={"ID":"c5ca672a-fa36-4182-8cac-59d4d8b992b6","Type":"ContainerStarted","Data":"41dcbf62abdd821af8a2af72b999cca8dcaa616550816de977a1b5725e0a194d"} Oct 30 02:45:20.222731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.222668 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh" event={"ID":"c5ca672a-fa36-4182-8cac-59d4d8b992b6","Type":"ContainerStarted","Data":"6c034cce34b20fa0b89f8cba96c9a226f80ffd605b27bf0e47b6aa9a6823449d"} Oct 30 02:45:20.225759 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.225728 5998 generic.go:356] "Generic (PLEG): container finished" podID="cce212c5-9e99-4e58-8dde-89b23f8e9e2e" containerID="8f30423f1a90e51527f5adfe78bd0619b81e4c78d6ea1cdecd278ed94de7a13e" exitCode=60 Oct 30 02:45:20.225872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.225758 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" event={"ID":"cce212c5-9e99-4e58-8dde-89b23f8e9e2e","Type":"ContainerDied","Data":"8f30423f1a90e51527f5adfe78bd0619b81e4c78d6ea1cdecd278ed94de7a13e"} Oct 30 02:45:20.226273 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.226154 5998 scope.go:122] "RemoveContainer" containerID="8f30423f1a90e51527f5adfe78bd0619b81e4c78d6ea1cdecd278ed94de7a13e" Oct 30 02:45:20.226583 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:20.226516 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g_odh-ai-gateway-infra(cce212c5-9e99-4e58-8dde-89b23f8e9e2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" podUID="cce212c5-9e99-4e58-8dde-89b23f8e9e2e" Oct 30 02:45:20.228708 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.227993 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" event={"ID":"c809da91-82e4-4c81-8644-d6409c5de91c","Type":"ContainerDied","Data":"631b0057a3116da442f9d9b88cbfc9f2250d9e61a30f00e3c0809f02e90d6152"} Oct 30 02:45:20.228708 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.228017 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld" Oct 30 02:45:20.230554 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.230538 5998 scope.go:122] "RemoveContainer" containerID="a85aa7897284fe1a3932c5d7d34a4e1d603c215a36943f3ef1fc3b2cd41375e0" Oct 30 02:45:20.230648 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.230620 5998 generic.go:356] "Generic (PLEG): container finished" podID="c560bec5-d0f1-4854-8ca0-24d44f73c750" containerID="7819c388f08e3745e3ba8a96ff0fac95f3a4ffcc1733f1a34164ce968ac68d09" exitCode=60 Oct 30 02:45:20.230725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.230654 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" event={"ID":"c560bec5-d0f1-4854-8ca0-24d44f73c750","Type":"ContainerDied","Data":"7819c388f08e3745e3ba8a96ff0fac95f3a4ffcc1733f1a34164ce968ac68d09"} Oct 30 02:45:20.231051 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.231022 5998 scope.go:122] "RemoveContainer" containerID="7819c388f08e3745e3ba8a96ff0fac95f3a4ffcc1733f1a34164ce968ac68d09" Oct 30 02:45:20.231353 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:20.231327 5998 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf_odh-ai-gateway-infra(c560bec5-d0f1-4854-8ca0-24d44f73c750)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" podUID="c560bec5-d0f1-4854-8ca0-24d44f73c750" Oct 30 02:45:20.254329 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.254288 5998 scope.go:122] "RemoveContainer" containerID="75e97c06d52cbf33f9ef64a178856f81ceb2c5033b03f3754c517606e0ade35c" Oct 30 02:45:20.271385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.271358 5998 scope.go:122] "RemoveContainer" containerID="a7dda9b140cdc81f84e40f0a55fc99bf7106064f6c5b35baf6939f7a35412e76" Oct 30 02:45:20.290595 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.290570 5998 scope.go:122] "RemoveContainer" containerID="3d117c0adf46209225d6ae72d3b8e98a942193c1a26fb7d57caf87ec81c9deb8" Oct 30 02:45:20.364220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.363839 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g"] Oct 30 02:45:20.394710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.394676 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" Oct 30 02:45:20.422717 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.422685 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" Oct 30 02:45:20.436507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.436476 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2"] Oct 30 02:45:20.454648 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.454614 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf"] Oct 30 02:45:20.521195 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.521159 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf"] Oct 30 02:45:20.542416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.542374 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8182c9bd-076d-4f26-8e5a-4596606d3d2d" path="/var/lib/kubelet/pods/8182c9bd-076d-4f26-8e5a-4596606d3d2d/volumes" Oct 30 02:45:20.542896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.542881 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="929aedab-934e-4cb3-b540-3dc89c8a974a" path="/var/lib/kubelet/pods/929aedab-934e-4cb3-b540-3dc89c8a974a/volumes" Oct 30 02:45:20.547783 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.547754 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9f59989-527d-411d-9fc0-ce758b48e18a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f59989-527d-411d-9fc0-ce758b48e18a-maas-api-service-ca\") pod \"b9f59989-527d-411d-9fc0-ce758b48e18a\" (UID: \"b9f59989-527d-411d-9fc0-ce758b48e18a\") " Oct 30 02:45:20.547925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.547902 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/662d5231-266e-4ff9-8e5a-4409da2c7d82-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/662d5231-266e-4ff9-8e5a-4409da2c7d82-maas-api-service-ca\") pod \"662d5231-266e-4ff9-8e5a-4409da2c7d82\" (UID: \"662d5231-266e-4ff9-8e5a-4409da2c7d82\") " Oct 30 02:45:20.548023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.547961 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f59989-527d-411d-9fc0-ce758b48e18a-kube-api-access-kwgst\" (UniqueName: \"kubernetes.io/projected/b9f59989-527d-411d-9fc0-ce758b48e18a-kube-api-access-kwgst\") pod \"b9f59989-527d-411d-9fc0-ce758b48e18a\" (UID: \"b9f59989-527d-411d-9fc0-ce758b48e18a\") " Oct 30 02:45:20.548023 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.548011 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/662d5231-266e-4ff9-8e5a-4409da2c7d82-kube-api-access-bzdbh\" (UniqueName: \"kubernetes.io/projected/662d5231-266e-4ff9-8e5a-4409da2c7d82-kube-api-access-bzdbh\") pod \"662d5231-266e-4ff9-8e5a-4409da2c7d82\" (UID: \"662d5231-266e-4ff9-8e5a-4409da2c7d82\") " Oct 30 02:45:20.548318 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.548289 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662d5231-266e-4ff9-8e5a-4409da2c7d82-maas-api-service-ca" pod "662d5231-266e-4ff9-8e5a-4409da2c7d82" (UID: "662d5231-266e-4ff9-8e5a-4409da2c7d82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:20.548817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.548471 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f59989-527d-411d-9fc0-ce758b48e18a-maas-api-service-ca" pod "b9f59989-527d-411d-9fc0-ce758b48e18a" (UID: "b9f59989-527d-411d-9fc0-ce758b48e18a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:20.550578 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.550533 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh" podStartSLOduration=2.550474833 podStartE2EDuration="2.550474833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:20.549500354 +0000 UTC m=+4550.128291436" watchObservedRunningTime="2026-10-30 02:45:20.550474833 +0000 UTC m=+4550.129266119" Oct 30 02:45:20.553482 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.550622 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662d5231-266e-4ff9-8e5a-4409da2c7d82-kube-api-access-bzdbh" pod "662d5231-266e-4ff9-8e5a-4409da2c7d82" (UID: "662d5231-266e-4ff9-8e5a-4409da2c7d82"). InnerVolumeSpecName "kube-api-access-bzdbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:20.553595 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.553511 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f59989-527d-411d-9fc0-ce758b48e18a-kube-api-access-kwgst" pod "b9f59989-527d-411d-9fc0-ce758b48e18a" (UID: "b9f59989-527d-411d-9fc0-ce758b48e18a"). InnerVolumeSpecName "kube-api-access-kwgst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:20.633015 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.632988 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld"] Oct 30 02:45:20.637945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.637917 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-594253-29888805-xwpld"] Oct 30 02:45:20.649717 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.649688 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f59989-527d-411d-9fc0-ce758b48e18a-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:20.649717 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.649717 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/662d5231-266e-4ff9-8e5a-4409da2c7d82-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:20.649897 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.649730 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwgst\" (UniqueName: \"kubernetes.io/projected/b9f59989-527d-411d-9fc0-ce758b48e18a-kube-api-access-kwgst\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:20.649897 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:20.649739 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzdbh\" (UniqueName: \"kubernetes.io/projected/662d5231-266e-4ff9-8e5a-4409da2c7d82-kube-api-access-bzdbh\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:21.108693 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.108657 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:21.255965 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.255936 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" Oct 30 02:45:21.256462 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.255941 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v" event={"ID":"662d5231-266e-4ff9-8e5a-4409da2c7d82","Type":"ContainerDied","Data":"fc8167c14cdad5cf51fd2fffd318df6d6aa211717921b24a096d5b52ce62d102"} Oct 30 02:45:21.256462 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.256075 5998 scope.go:122] "RemoveContainer" containerID="30e0fbce5e56cbadac5fc9c71c848e98652564457d2396b7db4a13dff0a201cc" Oct 30 02:45:21.260131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.259316 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" Oct 30 02:45:21.260131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.259334 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7" event={"ID":"b9f59989-527d-411d-9fc0-ce758b48e18a","Type":"ContainerDied","Data":"7a73e0cbeeedb9a643b486133a3b3ad15581d513802d17e50599d967a05260e4"} Oct 30 02:45:21.261893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.261865 5998 generic.go:356] "Generic (PLEG): container finished" podID="c5ca672a-fa36-4182-8cac-59d4d8b992b6" containerID="41dcbf62abdd821af8a2af72b999cca8dcaa616550816de977a1b5725e0a194d" exitCode=0 Oct 30 02:45:21.262033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.261997 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh" event={"ID":"c5ca672a-fa36-4182-8cac-59d4d8b992b6","Type":"ContainerDied","Data":"41dcbf62abdd821af8a2af72b999cca8dcaa616550816de977a1b5725e0a194d"} Oct 30 02:45:21.270736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.270708 5998 scope.go:122] "RemoveContainer" containerID="6a3e4e488f28cc938c9dfbf552dbbb657fd198b8f40d44ce54f9d9b0bf809d62" Oct 30 02:45:21.314115 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.309990 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v"] Oct 30 02:45:21.320166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.319464 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0fdc4e-29888805-s6n5v"] Oct 30 02:45:21.480769 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.480735 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" Oct 30 02:45:21.546256 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.546218 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7"] Oct 30 02:45:21.570921 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.569198 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" Oct 30 02:45:21.583639 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.583603 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-95f40f-29888805-zk2c7"] Oct 30 02:45:21.633256 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.633223 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" Oct 30 02:45:21.637300 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.637272 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" Oct 30 02:45:21.653574 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.653536 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:21.674898 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.674858 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce212c5-9e99-4e58-8dde-89b23f8e9e2e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce212c5-9e99-4e58-8dde-89b23f8e9e2e-maas-api-service-ca\") pod \"cce212c5-9e99-4e58-8dde-89b23f8e9e2e\" (UID: \"cce212c5-9e99-4e58-8dde-89b23f8e9e2e\") " Oct 30 02:45:21.675184 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.674952 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a0058da-63f6-48d6-9d96-316956c55dd4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0058da-63f6-48d6-9d96-316956c55dd4-maas-api-service-ca\") pod \"4a0058da-63f6-48d6-9d96-316956c55dd4\" (UID: \"4a0058da-63f6-48d6-9d96-316956c55dd4\") " Oct 30 02:45:21.675341 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.675235 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce212c5-9e99-4e58-8dde-89b23f8e9e2e-kube-api-access-m9dvp\" (UniqueName: \"kubernetes.io/projected/cce212c5-9e99-4e58-8dde-89b23f8e9e2e-kube-api-access-m9dvp\") pod \"cce212c5-9e99-4e58-8dde-89b23f8e9e2e\" (UID: \"cce212c5-9e99-4e58-8dde-89b23f8e9e2e\") " Oct 30 02:45:21.675341 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.675269 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0058da-63f6-48d6-9d96-316956c55dd4-kube-api-access-drbmc\" (UniqueName: \"kubernetes.io/projected/4a0058da-63f6-48d6-9d96-316956c55dd4-kube-api-access-drbmc\") pod \"4a0058da-63f6-48d6-9d96-316956c55dd4\" (UID: \"4a0058da-63f6-48d6-9d96-316956c55dd4\") " Oct 30 02:45:21.685335 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.681993 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0058da-63f6-48d6-9d96-316956c55dd4-maas-api-service-ca" pod "4a0058da-63f6-48d6-9d96-316956c55dd4" (UID: "4a0058da-63f6-48d6-9d96-316956c55dd4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:21.685335 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.682454 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce212c5-9e99-4e58-8dde-89b23f8e9e2e-maas-api-service-ca" pod "cce212c5-9e99-4e58-8dde-89b23f8e9e2e" (UID: "cce212c5-9e99-4e58-8dde-89b23f8e9e2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:21.689904 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.689833 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0058da-63f6-48d6-9d96-316956c55dd4-kube-api-access-drbmc" pod "4a0058da-63f6-48d6-9d96-316956c55dd4" (UID: "4a0058da-63f6-48d6-9d96-316956c55dd4"). InnerVolumeSpecName "kube-api-access-drbmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:21.691662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.691635 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce212c5-9e99-4e58-8dde-89b23f8e9e2e-kube-api-access-m9dvp" pod "cce212c5-9e99-4e58-8dde-89b23f8e9e2e" (UID: "cce212c5-9e99-4e58-8dde-89b23f8e9e2e"). InnerVolumeSpecName "kube-api-access-m9dvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:21.776952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.776918 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4-maas-api-service-ca\") pod \"e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4\" (UID: \"e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4\") " Oct 30 02:45:21.776952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.776967 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4-kube-api-access-rrpg4\" (UniqueName: \"kubernetes.io/projected/e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4-kube-api-access-rrpg4\") pod \"e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4\" (UID: \"e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4\") " Oct 30 02:45:21.777222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.777175 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c560bec5-d0f1-4854-8ca0-24d44f73c750-kube-api-access-8g7g5\" (UniqueName: \"kubernetes.io/projected/c560bec5-d0f1-4854-8ca0-24d44f73c750-kube-api-access-8g7g5\") pod \"c560bec5-d0f1-4854-8ca0-24d44f73c750\" (UID: \"c560bec5-d0f1-4854-8ca0-24d44f73c750\") " Oct 30 02:45:21.777313 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.777240 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c560bec5-d0f1-4854-8ca0-24d44f73c750-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c560bec5-d0f1-4854-8ca0-24d44f73c750-maas-api-service-ca\") pod \"c560bec5-d0f1-4854-8ca0-24d44f73c750\" (UID: \"c560bec5-d0f1-4854-8ca0-24d44f73c750\") " Oct 30 02:45:21.777775 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.777749 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9dvp\" (UniqueName: \"kubernetes.io/projected/cce212c5-9e99-4e58-8dde-89b23f8e9e2e-kube-api-access-m9dvp\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:21.777970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.777951 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drbmc\" (UniqueName: \"kubernetes.io/projected/4a0058da-63f6-48d6-9d96-316956c55dd4-kube-api-access-drbmc\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:21.778127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.778108 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce212c5-9e99-4e58-8dde-89b23f8e9e2e-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:21.778256 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.778237 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a0058da-63f6-48d6-9d96-316956c55dd4-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:21.778380 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.778130 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c560bec5-d0f1-4854-8ca0-24d44f73c750-maas-api-service-ca" pod "c560bec5-d0f1-4854-8ca0-24d44f73c750" (UID: "c560bec5-d0f1-4854-8ca0-24d44f73c750"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:21.778834 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.778530 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4-maas-api-service-ca" pod "e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4" (UID: "e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:21.779971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.779948 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c560bec5-d0f1-4854-8ca0-24d44f73c750-kube-api-access-8g7g5" pod "c560bec5-d0f1-4854-8ca0-24d44f73c750" (UID: "c560bec5-d0f1-4854-8ca0-24d44f73c750"). InnerVolumeSpecName "kube-api-access-8g7g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:21.780905 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.780885 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4-kube-api-access-rrpg4" pod "e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4" (UID: "e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4"). InnerVolumeSpecName "kube-api-access-rrpg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:21.879479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.879397 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g7g5\" (UniqueName: \"kubernetes.io/projected/c560bec5-d0f1-4854-8ca0-24d44f73c750-kube-api-access-8g7g5\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:21.879479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.879432 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c560bec5-d0f1-4854-8ca0-24d44f73c750-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:21.879479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.879447 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4-maas-api-service-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:21.879479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:21.879456 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrpg4\" (UniqueName: \"kubernetes.io/projected/e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4-kube-api-access-rrpg4\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:22.287271 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.287237 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" Oct 30 02:45:22.287751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.287265 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf" event={"ID":"c560bec5-d0f1-4854-8ca0-24d44f73c750","Type":"ContainerDied","Data":"c939db2a4a3c585a0811a88a3113eae677027ce569bc39e7a1f8cb34c84d4b33"} Oct 30 02:45:22.287751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.287372 5998 scope.go:122] "RemoveContainer" containerID="7819c388f08e3745e3ba8a96ff0fac95f3a4ffcc1733f1a34164ce968ac68d09" Oct 30 02:45:22.288843 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.288824 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" Oct 30 02:45:22.288941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.288877 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf" event={"ID":"e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4","Type":"ContainerDied","Data":"593d4e4b00b16f592deca6b9e34808ac28281dd00d10aff17b10c497b8639479"} Oct 30 02:45:22.291711 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.291673 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" event={"ID":"4a0058da-63f6-48d6-9d96-316956c55dd4","Type":"ContainerDied","Data":"5c8962ee9bace5e0c0724e3ad3e1162f9f54b9d5e0959f66479efca2b3f1eca1"} Oct 30 02:45:22.291711 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.291706 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2" Oct 30 02:45:22.294751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.294728 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" Oct 30 02:45:22.294862 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.294765 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g" event={"ID":"cce212c5-9e99-4e58-8dde-89b23f8e9e2e","Type":"ContainerDied","Data":"0033f12a0f303a9f459236ce4c1cd6073714ecd3c2d688c8d682ecae02d473b7"} Oct 30 02:45:22.307393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.307366 5998 scope.go:122] "RemoveContainer" containerID="a692cabf029e9dc783ec4c5a60c1012e18544e01cde29ab0d76e46b6978454e2" Oct 30 02:45:22.321034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.321006 5998 scope.go:122] "RemoveContainer" containerID="d8ba0ae6afdb5341163f0651c5cd7891e1478efe4332899ec3762a3eb6106f0e" Oct 30 02:45:22.334413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.334379 5998 scope.go:122] "RemoveContainer" containerID="8f30423f1a90e51527f5adfe78bd0619b81e4c78d6ea1cdecd278ed94de7a13e" Oct 30 02:45:22.383185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.381133 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g"] Oct 30 02:45:22.383185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.382985 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-01a9cd-29888805-rpf8g"] Oct 30 02:45:22.415429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.415372 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2"] Oct 30 02:45:22.422174 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.422131 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888805-sjxb2"] Oct 30 02:45:22.453924 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.453895 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh" Oct 30 02:45:22.461563 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.461529 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf"] Oct 30 02:45:22.465708 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.465679 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-24fb97-29888805-jl7pf"] Oct 30 02:45:22.497611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.497578 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf"] Oct 30 02:45:22.504034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.504005 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-719029-29888805-4sbtf"] Oct 30 02:45:22.542557 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.542475 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a0058da-63f6-48d6-9d96-316956c55dd4" path="/var/lib/kubelet/pods/4a0058da-63f6-48d6-9d96-316956c55dd4/volumes" Oct 30 02:45:22.543441 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.543413 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="662d5231-266e-4ff9-8e5a-4409da2c7d82" path="/var/lib/kubelet/pods/662d5231-266e-4ff9-8e5a-4409da2c7d82/volumes" Oct 30 02:45:22.544161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.544138 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9f59989-527d-411d-9fc0-ce758b48e18a" path="/var/lib/kubelet/pods/b9f59989-527d-411d-9fc0-ce758b48e18a/volumes" Oct 30 02:45:22.545034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.545011 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c560bec5-d0f1-4854-8ca0-24d44f73c750" path="/var/lib/kubelet/pods/c560bec5-d0f1-4854-8ca0-24d44f73c750/volumes" Oct 30 02:45:22.545511 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.545492 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c809da91-82e4-4c81-8644-d6409c5de91c" path="/var/lib/kubelet/pods/c809da91-82e4-4c81-8644-d6409c5de91c/volumes" Oct 30 02:45:22.545937 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.545917 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce212c5-9e99-4e58-8dde-89b23f8e9e2e" path="/var/lib/kubelet/pods/cce212c5-9e99-4e58-8dde-89b23f8e9e2e/volumes" Oct 30 02:45:22.546568 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.546547 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4" path="/var/lib/kubelet/pods/e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4/volumes" Oct 30 02:45:22.589674 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.589638 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ca672a-fa36-4182-8cac-59d4d8b992b6-kube-api-access-r8nvt\" (UniqueName: \"kubernetes.io/projected/c5ca672a-fa36-4182-8cac-59d4d8b992b6-kube-api-access-r8nvt\") pod \"c5ca672a-fa36-4182-8cac-59d4d8b992b6\" (UID: \"c5ca672a-fa36-4182-8cac-59d4d8b992b6\") " Oct 30 02:45:22.589872 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.589808 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ca672a-fa36-4182-8cac-59d4d8b992b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ca672a-fa36-4182-8cac-59d4d8b992b6-maas-api-openshift-ca\") pod \"c5ca672a-fa36-4182-8cac-59d4d8b992b6\" (UID: \"c5ca672a-fa36-4182-8cac-59d4d8b992b6\") " Oct 30 02:45:22.590022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.590004 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5ca672a-fa36-4182-8cac-59d4d8b992b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ca672a-fa36-4182-8cac-59d4d8b992b6-maas-api-xks-ca\") pod \"c5ca672a-fa36-4182-8cac-59d4d8b992b6\" (UID: \"c5ca672a-fa36-4182-8cac-59d4d8b992b6\") " Oct 30 02:45:22.590632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.590233 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ca672a-fa36-4182-8cac-59d4d8b992b6-maas-api-openshift-ca" pod "c5ca672a-fa36-4182-8cac-59d4d8b992b6" (UID: "c5ca672a-fa36-4182-8cac-59d4d8b992b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:22.590632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.590594 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5ca672a-fa36-4182-8cac-59d4d8b992b6-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:22.592317 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.592292 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ca672a-fa36-4182-8cac-59d4d8b992b6-maas-api-xks-ca" pod "c5ca672a-fa36-4182-8cac-59d4d8b992b6" (UID: "c5ca672a-fa36-4182-8cac-59d4d8b992b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:45:22.592432 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.592317 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ca672a-fa36-4182-8cac-59d4d8b992b6-kube-api-access-r8nvt" pod "c5ca672a-fa36-4182-8cac-59d4d8b992b6" (UID: "c5ca672a-fa36-4182-8cac-59d4d8b992b6"). InnerVolumeSpecName "kube-api-access-r8nvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:22.695851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.693139 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5ca672a-fa36-4182-8cac-59d4d8b992b6-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:22.695851 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.693179 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8nvt\" (UniqueName: \"kubernetes.io/projected/c5ca672a-fa36-4182-8cac-59d4d8b992b6-kube-api-access-r8nvt\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:22.773337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:22.773301 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:23.268867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:23.266210 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:23.316977 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:23.316939 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh" event={"ID":"c5ca672a-fa36-4182-8cac-59d4d8b992b6","Type":"ContainerDied","Data":"6c034cce34b20fa0b89f8cba96c9a226f80ffd605b27bf0e47b6aa9a6823449d"} Oct 30 02:45:23.317509 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:23.317008 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c034cce34b20fa0b89f8cba96c9a226f80ffd605b27bf0e47b6aa9a6823449d" Oct 30 02:45:23.317509 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:23.316949 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh" Oct 30 02:45:23.410516 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:23.410472 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:24.905991 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:24.905950 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:25.222325 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:25.222283 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:45:26.562427 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:26.562380 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:28.017327 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.017291 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:28.779334 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.779294 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:45:28.964244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964209 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv"] Oct 30 02:45:28.964713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964700 5998 cpu_manager.go:414] "containerMap: removing container" podUID="5bc5e6a0-5532-497a-8244-94b6768480f2" containerName="cleanup" Oct 30 02:45:28.964760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964715 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc5e6a0-5532-497a-8244-94b6768480f2" containerName="cleanup" Oct 30 02:45:28.964760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964731 5998 cpu_manager.go:414] "containerMap: removing container" podUID="cce212c5-9e99-4e58-8dde-89b23f8e9e2e" containerName="cleanup" Oct 30 02:45:28.964760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964737 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce212c5-9e99-4e58-8dde-89b23f8e9e2e" containerName="cleanup" Oct 30 02:45:28.964760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964745 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c5ca672a-fa36-4182-8cac-59d4d8b992b6" containerName="revoke-keys" Oct 30 02:45:28.964760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964750 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ca672a-fa36-4182-8cac-59d4d8b992b6" containerName="revoke-keys" Oct 30 02:45:28.964760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964761 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8182c9bd-076d-4f26-8e5a-4596606d3d2d" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964767 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8182c9bd-076d-4f26-8e5a-4596606d3d2d" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964774 5998 cpu_manager.go:414] "containerMap: removing container" podUID="b9f59989-527d-411d-9fc0-ce758b48e18a" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964779 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f59989-527d-411d-9fc0-ce758b48e18a" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964787 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c560bec5-d0f1-4854-8ca0-24d44f73c750" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964791 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c560bec5-d0f1-4854-8ca0-24d44f73c750" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964800 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c809da91-82e4-4c81-8644-d6409c5de91c" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964805 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c809da91-82e4-4c81-8644-d6409c5de91c" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964814 5998 cpu_manager.go:414] "containerMap: removing container" podUID="4a0058da-63f6-48d6-9d96-316956c55dd4" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964819 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0058da-63f6-48d6-9d96-316956c55dd4" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964827 5998 cpu_manager.go:414] "containerMap: removing container" podUID="e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964831 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964837 5998 cpu_manager.go:414] "containerMap: removing container" podUID="662d5231-266e-4ff9-8e5a-4409da2c7d82" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964843 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="662d5231-266e-4ff9-8e5a-4409da2c7d82" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964853 5998 cpu_manager.go:414] "containerMap: removing container" podUID="e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964858 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964864 5998 cpu_manager.go:414] "containerMap: removing container" podUID="662d5231-266e-4ff9-8e5a-4409da2c7d82" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964870 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="662d5231-266e-4ff9-8e5a-4409da2c7d82" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964878 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c809da91-82e4-4c81-8644-d6409c5de91c" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964883 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c809da91-82e4-4c81-8644-d6409c5de91c" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964891 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c560bec5-d0f1-4854-8ca0-24d44f73c750" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964895 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c560bec5-d0f1-4854-8ca0-24d44f73c750" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964904 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c809da91-82e4-4c81-8644-d6409c5de91c" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964909 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c809da91-82e4-4c81-8644-d6409c5de91c" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964914 5998 cpu_manager.go:414] "containerMap: removing container" podUID="4a0058da-63f6-48d6-9d96-316956c55dd4" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964919 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0058da-63f6-48d6-9d96-316956c55dd4" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964926 5998 cpu_manager.go:414] "containerMap: removing container" podUID="cce212c5-9e99-4e58-8dde-89b23f8e9e2e" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964930 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce212c5-9e99-4e58-8dde-89b23f8e9e2e" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964939 5998 cpu_manager.go:414] "containerMap: removing container" podUID="e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964944 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964953 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c560bec5-d0f1-4854-8ca0-24d44f73c750" containerName="cleanup" Oct 30 02:45:28.964952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964957 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c560bec5-d0f1-4854-8ca0-24d44f73c750" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964969 5998 cpu_manager.go:414] "containerMap: removing container" podUID="b9f59989-527d-411d-9fc0-ce758b48e18a" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.964974 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f59989-527d-411d-9fc0-ce758b48e18a" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965042 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce212c5-9e99-4e58-8dde-89b23f8e9e2e" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965049 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ca672a-fa36-4182-8cac-59d4d8b992b6" containerName="revoke-keys" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965056 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8182c9bd-076d-4f26-8e5a-4596606d3d2d" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965061 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f59989-527d-411d-9fc0-ce758b48e18a" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965067 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c560bec5-d0f1-4854-8ca0-24d44f73c750" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965073 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c809da91-82e4-4c81-8644-d6409c5de91c" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965081 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc5e6a0-5532-497a-8244-94b6768480f2" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965100 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0058da-63f6-48d6-9d96-316956c55dd4" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965106 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965112 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="662d5231-266e-4ff9-8e5a-4409da2c7d82" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965118 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965124 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="662d5231-266e-4ff9-8e5a-4409da2c7d82" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965129 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c809da91-82e4-4c81-8644-d6409c5de91c" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965134 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c560bec5-d0f1-4854-8ca0-24d44f73c750" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965139 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c809da91-82e4-4c81-8644-d6409c5de91c" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965145 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0058da-63f6-48d6-9d96-316956c55dd4" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965149 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce212c5-9e99-4e58-8dde-89b23f8e9e2e" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965154 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89e6f5d-8bc4-41f2-b6d1-0ae19c3536f4" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965160 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce212c5-9e99-4e58-8dde-89b23f8e9e2e" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965166 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0058da-63f6-48d6-9d96-316956c55dd4" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965172 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f59989-527d-411d-9fc0-ce758b48e18a" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965177 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="662d5231-266e-4ff9-8e5a-4409da2c7d82" containerName="cleanup" Oct 30 02:45:28.965900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.965183 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f59989-527d-411d-9fc0-ce758b48e18a" containerName="cleanup" Oct 30 02:45:28.970594 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.970573 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv" Oct 30 02:45:28.987426 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.987333 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:45:28.987668 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.987454 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:45:28.988011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.987984 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:45:28.989024 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.989000 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv"] Oct 30 02:45:28.995594 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:28.995567 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:45:29.079134 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:29.079013 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppvqc\" (UniqueName: \"kubernetes.io/projected/dffe7505-ba98-42c5-aba9-0ef9539b16e4-kube-api-access-ppvqc\") pod \"maas-api-revoke-sub-fa868ea5-mkpcv\" (UID: \"dffe7505-ba98-42c5-aba9-0ef9539b16e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv" Oct 30 02:45:29.079134 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:29.079064 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffe7505-ba98-42c5-aba9-0ef9539b16e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa868ea5-mkpcv\" (UID: \"dffe7505-ba98-42c5-aba9-0ef9539b16e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv" Oct 30 02:45:29.079718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:29.079196 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffe7505-ba98-42c5-aba9-0ef9539b16e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa868ea5-mkpcv\" (UID: \"dffe7505-ba98-42c5-aba9-0ef9539b16e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv" Oct 30 02:45:29.180479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:29.180437 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffe7505-ba98-42c5-aba9-0ef9539b16e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa868ea5-mkpcv\" (UID: \"dffe7505-ba98-42c5-aba9-0ef9539b16e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv" Oct 30 02:45:29.180686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:29.180667 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppvqc\" (UniqueName: \"kubernetes.io/projected/dffe7505-ba98-42c5-aba9-0ef9539b16e4-kube-api-access-ppvqc\") pod \"maas-api-revoke-sub-fa868ea5-mkpcv\" (UID: \"dffe7505-ba98-42c5-aba9-0ef9539b16e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv" Oct 30 02:45:29.180757 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:29.180711 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffe7505-ba98-42c5-aba9-0ef9539b16e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa868ea5-mkpcv\" (UID: \"dffe7505-ba98-42c5-aba9-0ef9539b16e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv" Oct 30 02:45:29.181422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:29.181394 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffe7505-ba98-42c5-aba9-0ef9539b16e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa868ea5-mkpcv\" (UID: \"dffe7505-ba98-42c5-aba9-0ef9539b16e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv" Oct 30 02:45:29.182918 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:29.182895 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffe7505-ba98-42c5-aba9-0ef9539b16e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa868ea5-mkpcv\" (UID: \"dffe7505-ba98-42c5-aba9-0ef9539b16e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv" Oct 30 02:45:29.217955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:29.217917 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppvqc\" (UniqueName: \"kubernetes.io/projected/dffe7505-ba98-42c5-aba9-0ef9539b16e4-kube-api-access-ppvqc\") pod \"maas-api-revoke-sub-fa868ea5-mkpcv\" (UID: \"dffe7505-ba98-42c5-aba9-0ef9539b16e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv" Oct 30 02:45:29.284968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:29.284901 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv" Oct 30 02:45:29.530691 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:29.530652 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:29.729931 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:29.729882 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:29.739926 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:29.739893 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv"] Oct 30 02:45:29.744797 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:45:29.743143 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffe7505_ba98_42c5_aba9_0ef9539b16e4.slice/crio-99683b2cb51b0553f60784e76385d22b083fca1009be121696c9eecddf44bf58 WatchSource:0}: Error finding container 99683b2cb51b0553f60784e76385d22b083fca1009be121696c9eecddf44bf58: Status 404 returned error can't find the container with id 99683b2cb51b0553f60784e76385d22b083fca1009be121696c9eecddf44bf58 Oct 30 02:45:29.994068 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:29.993909 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:30.384443 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:30.384349 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv" event={"ID":"dffe7505-ba98-42c5-aba9-0ef9539b16e4","Type":"ContainerStarted","Data":"78a9a6e58ed31ddf6775eb00e4d08ae9e3effe18e7c7563d8420e0db3202835e"} Oct 30 02:45:30.384443 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:30.384403 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv" event={"ID":"dffe7505-ba98-42c5-aba9-0ef9539b16e4","Type":"ContainerStarted","Data":"99683b2cb51b0553f60784e76385d22b083fca1009be121696c9eecddf44bf58"} Oct 30 02:45:30.422998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:30.422935 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv" podStartSLOduration=2.422922664 podStartE2EDuration="2.422922664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:45:30.421438626 +0000 UTC m=+4560.000229708" watchObservedRunningTime="2026-10-30 02:45:30.422922664 +0000 UTC m=+4560.001713744" Oct 30 02:45:31.014858 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:31.014757 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:31.126417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:31.126354 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:31.392843 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:31.392744 5998 generic.go:356] "Generic (PLEG): container finished" podID="dffe7505-ba98-42c5-aba9-0ef9539b16e4" containerID="78a9a6e58ed31ddf6775eb00e4d08ae9e3effe18e7c7563d8420e0db3202835e" exitCode=0 Oct 30 02:45:31.393340 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:31.392831 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv" event={"ID":"dffe7505-ba98-42c5-aba9-0ef9539b16e4","Type":"ContainerDied","Data":"78a9a6e58ed31ddf6775eb00e4d08ae9e3effe18e7c7563d8420e0db3202835e"} Oct 30 02:45:31.785796 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:31.785757 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:32.382706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:32.382654 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:32.549857 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:32.549830 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv" Oct 30 02:45:32.673303 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:32.673271 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dffe7505-ba98-42c5-aba9-0ef9539b16e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffe7505-ba98-42c5-aba9-0ef9539b16e4-maas-api-xks-ca\") pod \"dffe7505-ba98-42c5-aba9-0ef9539b16e4\" (UID: \"dffe7505-ba98-42c5-aba9-0ef9539b16e4\") " Oct 30 02:45:32.673509 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:32.673329 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dffe7505-ba98-42c5-aba9-0ef9539b16e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffe7505-ba98-42c5-aba9-0ef9539b16e4-maas-api-openshift-ca\") pod \"dffe7505-ba98-42c5-aba9-0ef9539b16e4\" (UID: \"dffe7505-ba98-42c5-aba9-0ef9539b16e4\") " Oct 30 02:45:32.673509 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:32.673383 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dffe7505-ba98-42c5-aba9-0ef9539b16e4-kube-api-access-ppvqc\" (UniqueName: \"kubernetes.io/projected/dffe7505-ba98-42c5-aba9-0ef9539b16e4-kube-api-access-ppvqc\") pod \"dffe7505-ba98-42c5-aba9-0ef9539b16e4\" (UID: \"dffe7505-ba98-42c5-aba9-0ef9539b16e4\") " Oct 30 02:45:32.673772 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:32.673747 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffe7505-ba98-42c5-aba9-0ef9539b16e4-maas-api-openshift-ca" pod "dffe7505-ba98-42c5-aba9-0ef9539b16e4" (UID: "dffe7505-ba98-42c5-aba9-0ef9539b16e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:32.675457 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:32.675435 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dffe7505-ba98-42c5-aba9-0ef9539b16e4-maas-api-xks-ca" pod "dffe7505-ba98-42c5-aba9-0ef9539b16e4" (UID: "dffe7505-ba98-42c5-aba9-0ef9539b16e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:45:32.675523 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:32.675505 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffe7505-ba98-42c5-aba9-0ef9539b16e4-kube-api-access-ppvqc" pod "dffe7505-ba98-42c5-aba9-0ef9539b16e4" (UID: "dffe7505-ba98-42c5-aba9-0ef9539b16e4"). InnerVolumeSpecName "kube-api-access-ppvqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:32.775034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:32.774996 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppvqc\" (UniqueName: \"kubernetes.io/projected/dffe7505-ba98-42c5-aba9-0ef9539b16e4-kube-api-access-ppvqc\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:32.775034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:32.775027 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffe7505-ba98-42c5-aba9-0ef9539b16e4-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:32.775034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:32.775036 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffe7505-ba98-42c5-aba9-0ef9539b16e4-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:33.412267 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:33.412233 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv" Oct 30 02:45:33.412482 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:33.412272 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv" event={"ID":"dffe7505-ba98-42c5-aba9-0ef9539b16e4","Type":"ContainerDied","Data":"99683b2cb51b0553f60784e76385d22b083fca1009be121696c9eecddf44bf58"} Oct 30 02:45:33.412482 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:33.412338 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99683b2cb51b0553f60784e76385d22b083fca1009be121696c9eecddf44bf58" Oct 30 02:45:33.419277 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:33.419235 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:33.759267 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:33.759153 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:33.864824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:33.864781 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:35.078781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:35.078741 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:35.835112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:35.834121 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:35.957353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:35.957310 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:36.857063 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:36.857023 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:37.581808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:37.581772 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:38.544373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:38.544331 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:39.347064 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:39.347028 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:40.073318 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:40.073284 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:40.721490 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:40.721450 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:41.332228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:41.332181 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r"] Oct 30 02:45:41.362480 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:41.362443 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:41.500125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:41.496820 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:42.336689 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:42.336650 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:43.933398 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:43.933364 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:44.581896 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:44.581853 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:45.497646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:45.497610 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:46.514745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:46.514707 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:46.627164 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:46.627125 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:47.163182 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:47.163143 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:47.325508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:47.325467 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:47.688136 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:47.688079 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7"] Oct 30 02:45:47.689361 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:47.689304 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" podUID="7076db80-f9a3-423f-b198-f0a7aba80bd7" containerName="istio-proxy" containerID="cri-o://99e92e7f5539f677d3f2780853aa31a4e19ea6f62559d08218f1717923b078d3" gracePeriod=30 Oct 30 02:45:47.724194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:47.724158 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:48.467162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:48.467125 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:49.055051 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:49.055010 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:50.576024 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:50.575989 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn"] Oct 30 02:45:50.576630 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:50.576053 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs"] Oct 30 02:45:50.576998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:50.576964 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" podUID="42fa7bf8-25e0-4bd8-b751-243b12d986b3" containerName="istio-proxy" containerID="cri-o://4b7c66f045a1d2292dd404ae604ee47c603d9e0acd3a591c7aa5f3243f2a2dd9" gracePeriod=30 Oct 30 02:45:50.577407 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:50.577379 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" podUID="c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" containerName="istio-proxy" containerID="cri-o://0470cb631abc84e26e55ca4d39d66bc5d8c35568914bb2540fdb1346db1d4588" gracePeriod=30 Oct 30 02:45:50.606476 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:50.606441 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m"] Oct 30 02:45:50.606721 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:50.606693 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" podUID="acb7072a-7343-4122-9472-93c1a658cb02" containerName="istio-proxy" containerID="cri-o://8b0c1bb4a6b32938478201745bc90f2d0ffbcfe624caf4f6a1be410f982de16b" gracePeriod=30 Oct 30 02:45:50.643600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:50.643559 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm"] Oct 30 02:45:50.644108 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:50.644053 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" podUID="94e4cea7-0d7d-46b1-810b-4567363be4eb" containerName="istio-proxy" containerID="cri-o://dd9f169eaba798b00bc61f97c0fdb48011ba289ff77db7b62c63c688eb87fcc5" gracePeriod=30 Oct 30 02:45:50.892656 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:50.892564 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:52.552419 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:52.552378 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:52.969204 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:52.969178 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:45:53.081494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.081394 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-workload-socket\") pod \"7076db80-f9a3-423f-b198-f0a7aba80bd7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " Oct 30 02:45:53.081660 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.081573 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-token\" (UniqueName: \"kubernetes.io/projected/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-token\") pod \"7076db80-f9a3-423f-b198-f0a7aba80bd7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " Oct 30 02:45:53.081660 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.081599 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-workload-certs\") pod \"7076db80-f9a3-423f-b198-f0a7aba80bd7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " Oct 30 02:45:53.081660 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.081619 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7076db80-f9a3-423f-b198-f0a7aba80bd7-kube-api-access-qnbl2\" (UniqueName: \"kubernetes.io/projected/7076db80-f9a3-423f-b198-f0a7aba80bd7-kube-api-access-qnbl2\") pod \"7076db80-f9a3-423f-b198-f0a7aba80bd7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " Oct 30 02:45:53.081834 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.081667 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-credential-socket\") pod \"7076db80-f9a3-423f-b198-f0a7aba80bd7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " Oct 30 02:45:53.081834 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.081690 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-workload-socket" pod "7076db80-f9a3-423f-b198-f0a7aba80bd7" (UID: "7076db80-f9a3-423f-b198-f0a7aba80bd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:53.081834 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.081701 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-data\") pod \"7076db80-f9a3-423f-b198-f0a7aba80bd7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " Oct 30 02:45:53.081834 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.081762 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-podinfo\") pod \"7076db80-f9a3-423f-b198-f0a7aba80bd7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " Oct 30 02:45:53.081834 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.081817 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-envoy\") pod \"7076db80-f9a3-423f-b198-f0a7aba80bd7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " Oct 30 02:45:53.082104 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.081890 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7076db80-f9a3-423f-b198-f0a7aba80bd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7076db80-f9a3-423f-b198-f0a7aba80bd7-istiod-ca-cert\") pod \"7076db80-f9a3-423f-b198-f0a7aba80bd7\" (UID: \"7076db80-f9a3-423f-b198-f0a7aba80bd7\") " Oct 30 02:45:53.082104 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.081925 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-workload-certs" pod "7076db80-f9a3-423f-b198-f0a7aba80bd7" (UID: "7076db80-f9a3-423f-b198-f0a7aba80bd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:53.082104 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.081986 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-data" pod "7076db80-f9a3-423f-b198-f0a7aba80bd7" (UID: "7076db80-f9a3-423f-b198-f0a7aba80bd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:53.082104 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.082046 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-credential-socket" pod "7076db80-f9a3-423f-b198-f0a7aba80bd7" (UID: "7076db80-f9a3-423f-b198-f0a7aba80bd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:53.082503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.082452 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7076db80-f9a3-423f-b198-f0a7aba80bd7-istiod-ca-cert" pod "7076db80-f9a3-423f-b198-f0a7aba80bd7" (UID: "7076db80-f9a3-423f-b198-f0a7aba80bd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:53.082503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.082480 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:53.082503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.082501 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:53.082743 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.082514 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:53.082743 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.082526 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:53.084262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.084139 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-token" pod "7076db80-f9a3-423f-b198-f0a7aba80bd7" (UID: "7076db80-f9a3-423f-b198-f0a7aba80bd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:53.084455 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.084295 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-envoy" pod "7076db80-f9a3-423f-b198-f0a7aba80bd7" (UID: "7076db80-f9a3-423f-b198-f0a7aba80bd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:53.084659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.084620 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-podinfo" pod "7076db80-f9a3-423f-b198-f0a7aba80bd7" (UID: "7076db80-f9a3-423f-b198-f0a7aba80bd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:45:53.084751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.084666 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7076db80-f9a3-423f-b198-f0a7aba80bd7-kube-api-access-qnbl2" pod "7076db80-f9a3-423f-b198-f0a7aba80bd7" (UID: "7076db80-f9a3-423f-b198-f0a7aba80bd7"). InnerVolumeSpecName "kube-api-access-qnbl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:53.183482 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.183451 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7076db80-f9a3-423f-b198-f0a7aba80bd7-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:53.183482 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.183478 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:53.183706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.183488 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnbl2\" (UniqueName: \"kubernetes.io/projected/7076db80-f9a3-423f-b198-f0a7aba80bd7-kube-api-access-qnbl2\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:53.183706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.183503 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:53.183706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.183513 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7076db80-f9a3-423f-b198-f0a7aba80bd7-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:53.629264 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.629222 5998 generic.go:356] "Generic (PLEG): container finished" podID="7076db80-f9a3-423f-b198-f0a7aba80bd7" containerID="99e92e7f5539f677d3f2780853aa31a4e19ea6f62559d08218f1717923b078d3" exitCode=0 Oct 30 02:45:53.629751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.629302 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" Oct 30 02:45:53.629751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.629324 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" event={"ID":"7076db80-f9a3-423f-b198-f0a7aba80bd7","Type":"ContainerDied","Data":"99e92e7f5539f677d3f2780853aa31a4e19ea6f62559d08218f1717923b078d3"} Oct 30 02:45:53.629751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.629395 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7" event={"ID":"7076db80-f9a3-423f-b198-f0a7aba80bd7","Type":"ContainerDied","Data":"61da4ffb1bb1590733c271d8a64fab8ba8ac9b855b2902a58e30dde4b3cc5000"} Oct 30 02:45:53.629751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.629421 5998 scope.go:122] "RemoveContainer" containerID="99e92e7f5539f677d3f2780853aa31a4e19ea6f62559d08218f1717923b078d3" Oct 30 02:45:53.642691 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.642662 5998 scope.go:122] "RemoveContainer" containerID="912eab5621d876448ae3aa2a0207002feac417c12e21aa8cae2226c48ff37021" Oct 30 02:45:53.655726 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.655705 5998 scope.go:122] "RemoveContainer" containerID="99e92e7f5539f677d3f2780853aa31a4e19ea6f62559d08218f1717923b078d3" Oct 30 02:45:53.656137 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:53.656075 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99e92e7f5539f677d3f2780853aa31a4e19ea6f62559d08218f1717923b078d3\": container with ID starting with 99e92e7f5539f677d3f2780853aa31a4e19ea6f62559d08218f1717923b078d3 not found: ID does not exist" containerID="99e92e7f5539f677d3f2780853aa31a4e19ea6f62559d08218f1717923b078d3" Oct 30 02:45:53.656236 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.656152 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e92e7f5539f677d3f2780853aa31a4e19ea6f62559d08218f1717923b078d3"} err="failed to get container status \"99e92e7f5539f677d3f2780853aa31a4e19ea6f62559d08218f1717923b078d3\": rpc error: code = NotFound desc = could not find container \"99e92e7f5539f677d3f2780853aa31a4e19ea6f62559d08218f1717923b078d3\": container with ID starting with 99e92e7f5539f677d3f2780853aa31a4e19ea6f62559d08218f1717923b078d3 not found: ID does not exist" Oct 30 02:45:53.656236 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.656178 5998 scope.go:122] "RemoveContainer" containerID="912eab5621d876448ae3aa2a0207002feac417c12e21aa8cae2226c48ff37021" Oct 30 02:45:53.656548 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:53.656520 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"912eab5621d876448ae3aa2a0207002feac417c12e21aa8cae2226c48ff37021\": container with ID starting with 912eab5621d876448ae3aa2a0207002feac417c12e21aa8cae2226c48ff37021 not found: ID does not exist" containerID="912eab5621d876448ae3aa2a0207002feac417c12e21aa8cae2226c48ff37021" Oct 30 02:45:53.656598 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.656558 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"912eab5621d876448ae3aa2a0207002feac417c12e21aa8cae2226c48ff37021"} err="failed to get container status \"912eab5621d876448ae3aa2a0207002feac417c12e21aa8cae2226c48ff37021\": rpc error: code = NotFound desc = could not find container \"912eab5621d876448ae3aa2a0207002feac417c12e21aa8cae2226c48ff37021\": container with ID starting with 912eab5621d876448ae3aa2a0207002feac417c12e21aa8cae2226c48ff37021 not found: ID does not exist" Oct 30 02:45:53.677907 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.677875 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7"] Oct 30 02:45:53.687981 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:53.687951 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-95f40f-openshift-default-c68d485cc-z2ml7"] Oct 30 02:45:54.169781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:54.169728 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:54.542332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:54.542231 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7076db80-f9a3-423f-b198-f0a7aba80bd7" path="/var/lib/kubelet/pods/7076db80-f9a3-423f-b198-f0a7aba80bd7/volumes" Oct 30 02:45:54.543189 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:54.543159 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf"] Oct 30 02:45:54.588711 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:54.588678 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:55.116602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:55.116554 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:55.219138 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:55.218724 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:45:55.829488 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:55.829331 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:55.943915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:55.943823 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:55.945418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:55.945361 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:45:55.963916 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:55.963884 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:45:55.979633 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:55.979606 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:45:56.020845 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.020809 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-workload-certs\") pod \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " Oct 30 02:45:56.021041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.020859 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-workload-certs\") pod \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " Oct 30 02:45:56.021041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.020897 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-data\") pod \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " Oct 30 02:45:56.021041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.020936 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-istio-envoy\") pod \"acb7072a-7343-4122-9472-93c1a658cb02\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " Oct 30 02:45:56.021041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.020972 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istiod-ca-cert\") pod \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " Oct 30 02:45:56.021041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021025 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-token\" (UniqueName: \"kubernetes.io/projected/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-token\") pod \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " Oct 30 02:45:56.021410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021055 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-data\") pod \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " Oct 30 02:45:56.021410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021108 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-kube-api-access-xhm55\" (UniqueName: \"kubernetes.io/projected/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-kube-api-access-xhm55\") pod \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " Oct 30 02:45:56.021410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021170 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istiod-ca-cert\") pod \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " Oct 30 02:45:56.021410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021204 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-token\" (UniqueName: \"kubernetes.io/projected/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-token\") pod \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " Oct 30 02:45:56.021410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021239 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-envoy\") pod \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " Oct 30 02:45:56.021410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021270 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-workload-socket\") pod \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " Oct 30 02:45:56.021410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021290 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-data" pod "42fa7bf8-25e0-4bd8-b751-243b12d986b3" (UID: "42fa7bf8-25e0-4bd8-b751-243b12d986b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.021410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021309 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-workload-socket\") pod \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " Oct 30 02:45:56.021410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021324 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-workload-certs" pod "c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" (UID: "c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.021410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021352 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-podinfo\") pod \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " Oct 30 02:45:56.021410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021412 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acb7072a-7343-4122-9472-93c1a658cb02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb7072a-7343-4122-9472-93c1a658cb02-istio-podinfo\") pod \"acb7072a-7343-4122-9472-93c1a658cb02\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021448 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-podinfo\") pod \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021691 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-credential-socket\") pod \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021736 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-credential-socket\") pod \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021801 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-workload-certs\") pod \"acb7072a-7343-4122-9472-93c1a658cb02\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021840 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-credential-socket\") pod \"acb7072a-7343-4122-9472-93c1a658cb02\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021886 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb7072a-7343-4122-9472-93c1a658cb02-istio-token\" (UniqueName: \"kubernetes.io/projected/acb7072a-7343-4122-9472-93c1a658cb02-istio-token\") pod \"acb7072a-7343-4122-9472-93c1a658cb02\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021923 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istiod-ca-cert" pod "42fa7bf8-25e0-4bd8-b751-243b12d986b3" (UID: "42fa7bf8-25e0-4bd8-b751-243b12d986b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.021965 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acb7072a-7343-4122-9472-93c1a658cb02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb7072a-7343-4122-9472-93c1a658cb02-istiod-ca-cert\") pod \"acb7072a-7343-4122-9472-93c1a658cb02\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.022002 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-workload-socket\") pod \"acb7072a-7343-4122-9472-93c1a658cb02\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.022056 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42fa7bf8-25e0-4bd8-b751-243b12d986b3-kube-api-access-5l47m\" (UniqueName: \"kubernetes.io/projected/42fa7bf8-25e0-4bd8-b751-243b12d986b3-kube-api-access-5l47m\") pod \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\" (UID: \"42fa7bf8-25e0-4bd8-b751-243b12d986b3\") " Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.022119 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-envoy\") pod \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\" (UID: \"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b\") " Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.022173 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-istio-data\") pod \"acb7072a-7343-4122-9472-93c1a658cb02\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.022221 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb7072a-7343-4122-9472-93c1a658cb02-kube-api-access-7zgxj\" (UniqueName: \"kubernetes.io/projected/acb7072a-7343-4122-9472-93c1a658cb02-kube-api-access-7zgxj\") pod \"acb7072a-7343-4122-9472-93c1a658cb02\" (UID: \"acb7072a-7343-4122-9472-93c1a658cb02\") " Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.022292 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-workload-socket" pod "c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" (UID: "c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.022628 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-workload-certs" pod "42fa7bf8-25e0-4bd8-b751-243b12d986b3" (UID: "42fa7bf8-25e0-4bd8-b751-243b12d986b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.022746 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-workload-socket" pod "42fa7bf8-25e0-4bd8-b751-243b12d986b3" (UID: "42fa7bf8-25e0-4bd8-b751-243b12d986b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.022880 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.022898 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.022912 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.022924 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.022938 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.022987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.022950 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.023927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.023303 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istiod-ca-cert" pod "c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" (UID: "c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:56.023927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.023520 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-data" pod "c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" (UID: "c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.024591 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.024567 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-credential-socket" pod "c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" (UID: "c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.024802 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.024781 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-credential-socket" pod "42fa7bf8-25e0-4bd8-b751-243b12d986b3" (UID: "42fa7bf8-25e0-4bd8-b751-243b12d986b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.025032 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.025010 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-workload-certs" pod "acb7072a-7343-4122-9472-93c1a658cb02" (UID: "acb7072a-7343-4122-9472-93c1a658cb02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.025308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.025280 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-credential-socket" pod "acb7072a-7343-4122-9472-93c1a658cb02" (UID: "acb7072a-7343-4122-9472-93c1a658cb02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.025410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.025387 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb7072a-7343-4122-9472-93c1a658cb02-istiod-ca-cert" pod "acb7072a-7343-4122-9472-93c1a658cb02" (UID: "acb7072a-7343-4122-9472-93c1a658cb02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:56.025524 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.025501 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-workload-socket" pod "acb7072a-7343-4122-9472-93c1a658cb02" (UID: "acb7072a-7343-4122-9472-93c1a658cb02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.025733 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.025711 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-istio-data" pod "acb7072a-7343-4122-9472-93c1a658cb02" (UID: "acb7072a-7343-4122-9472-93c1a658cb02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.027492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.027455 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-istio-envoy" pod "acb7072a-7343-4122-9472-93c1a658cb02" (UID: "acb7072a-7343-4122-9472-93c1a658cb02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.027724 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.027689 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-podinfo" pod "42fa7bf8-25e0-4bd8-b751-243b12d986b3" (UID: "42fa7bf8-25e0-4bd8-b751-243b12d986b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:45:56.035510 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.035476 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acb7072a-7343-4122-9472-93c1a658cb02-istio-podinfo" pod "acb7072a-7343-4122-9472-93c1a658cb02" (UID: "acb7072a-7343-4122-9472-93c1a658cb02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:45:56.035625 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.035505 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb7072a-7343-4122-9472-93c1a658cb02-kube-api-access-7zgxj" pod "acb7072a-7343-4122-9472-93c1a658cb02" (UID: "acb7072a-7343-4122-9472-93c1a658cb02"). InnerVolumeSpecName "kube-api-access-7zgxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:56.035625 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.035482 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-kube-api-access-xhm55" pod "c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" (UID: "c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b"). InnerVolumeSpecName "kube-api-access-xhm55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:56.035625 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.035512 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-token" pod "c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" (UID: "c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:56.035625 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.035538 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42fa7bf8-25e0-4bd8-b751-243b12d986b3-kube-api-access-5l47m" pod "42fa7bf8-25e0-4bd8-b751-243b12d986b3" (UID: "42fa7bf8-25e0-4bd8-b751-243b12d986b3"). InnerVolumeSpecName "kube-api-access-5l47m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:56.035625 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.035554 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb7072a-7343-4122-9472-93c1a658cb02-istio-token" pod "acb7072a-7343-4122-9472-93c1a658cb02" (UID: "acb7072a-7343-4122-9472-93c1a658cb02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:56.035820 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.035643 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-podinfo" pod "c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" (UID: "c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:45:56.035820 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.035655 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-envoy" pod "42fa7bf8-25e0-4bd8-b751-243b12d986b3" (UID: "42fa7bf8-25e0-4bd8-b751-243b12d986b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.035820 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.035658 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-token" pod "42fa7bf8-25e0-4bd8-b751-243b12d986b3" (UID: "42fa7bf8-25e0-4bd8-b751-243b12d986b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:56.037218 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.037191 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-envoy" pod "c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" (UID: "c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.055562 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.055381 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:56.124191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124072 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l47m\" (UniqueName: \"kubernetes.io/projected/42fa7bf8-25e0-4bd8-b751-243b12d986b3-kube-api-access-5l47m\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.124191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124140 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.124191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124158 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.124191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124167 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zgxj\" (UniqueName: \"kubernetes.io/projected/acb7072a-7343-4122-9472-93c1a658cb02-kube-api-access-7zgxj\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.124191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124175 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.124191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124184 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.124191 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124191 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.125017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124200 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.125017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124209 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhm55\" (UniqueName: \"kubernetes.io/projected/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-kube-api-access-xhm55\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.125017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124222 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.125017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124234 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.125017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124241 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.125017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124249 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb7072a-7343-4122-9472-93c1a658cb02-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.125017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124256 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42fa7bf8-25e0-4bd8-b751-243b12d986b3-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.125017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124264 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42fa7bf8-25e0-4bd8-b751-243b12d986b3-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.125017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124274 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.125017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124282 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.125017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124291 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.125017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124306 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb7072a-7343-4122-9472-93c1a658cb02-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.125017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124314 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb7072a-7343-4122-9472-93c1a658cb02-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.125017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.124321 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb7072a-7343-4122-9472-93c1a658cb02-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.357771 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.357602 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" podUID="13bc0eed-818a-4ef8-bc7a-6d9a2aed513f" containerName="main" containerID="cri-o://1c6c1bd95fe09c8b35543adea05056e16b5b0c674a9543da57ac934e02a2a234" gracePeriod=45 Oct 30 02:45:56.472448 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.472422 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:45:56.534750 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.534722 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-token\" (UniqueName: \"kubernetes.io/projected/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-token\") pod \"94e4cea7-0d7d-46b1-810b-4567363be4eb\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " Oct 30 02:45:56.534901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.534786 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-workload-certs\") pod \"94e4cea7-0d7d-46b1-810b-4567363be4eb\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " Oct 30 02:45:56.534901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.534857 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e4cea7-0d7d-46b1-810b-4567363be4eb-kube-api-access-dwd2r\" (UniqueName: \"kubernetes.io/projected/94e4cea7-0d7d-46b1-810b-4567363be4eb-kube-api-access-dwd2r\") pod \"94e4cea7-0d7d-46b1-810b-4567363be4eb\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " Oct 30 02:45:56.535024 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.534930 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-data\") pod \"94e4cea7-0d7d-46b1-810b-4567363be4eb\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " Oct 30 02:45:56.535024 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.534999 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-envoy\") pod \"94e4cea7-0d7d-46b1-810b-4567363be4eb\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " Oct 30 02:45:56.535234 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.535044 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-workload-socket\") pod \"94e4cea7-0d7d-46b1-810b-4567363be4eb\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " Oct 30 02:45:56.535234 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.535130 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e4cea7-0d7d-46b1-810b-4567363be4eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e4cea7-0d7d-46b1-810b-4567363be4eb-istiod-ca-cert\") pod \"94e4cea7-0d7d-46b1-810b-4567363be4eb\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " Oct 30 02:45:56.535234 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.535220 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-podinfo\") pod \"94e4cea7-0d7d-46b1-810b-4567363be4eb\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " Oct 30 02:45:56.535393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.535249 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-credential-socket\") pod \"94e4cea7-0d7d-46b1-810b-4567363be4eb\" (UID: \"94e4cea7-0d7d-46b1-810b-4567363be4eb\") " Oct 30 02:45:56.537345 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.536009 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-data" pod "94e4cea7-0d7d-46b1-810b-4567363be4eb" (UID: "94e4cea7-0d7d-46b1-810b-4567363be4eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.537345 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.536123 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-credential-socket" pod "94e4cea7-0d7d-46b1-810b-4567363be4eb" (UID: "94e4cea7-0d7d-46b1-810b-4567363be4eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.537345 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.536474 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e4cea7-0d7d-46b1-810b-4567363be4eb-istiod-ca-cert" pod "94e4cea7-0d7d-46b1-810b-4567363be4eb" (UID: "94e4cea7-0d7d-46b1-810b-4567363be4eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:45:56.537345 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.536583 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-workload-socket" pod "94e4cea7-0d7d-46b1-810b-4567363be4eb" (UID: "94e4cea7-0d7d-46b1-810b-4567363be4eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.537345 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.537283 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-workload-certs" pod "94e4cea7-0d7d-46b1-810b-4567363be4eb" (UID: "94e4cea7-0d7d-46b1-810b-4567363be4eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.539155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.539130 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-podinfo" pod "94e4cea7-0d7d-46b1-810b-4567363be4eb" (UID: "94e4cea7-0d7d-46b1-810b-4567363be4eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:45:56.539287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.539173 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e4cea7-0d7d-46b1-810b-4567363be4eb-kube-api-access-dwd2r" pod "94e4cea7-0d7d-46b1-810b-4567363be4eb" (UID: "94e4cea7-0d7d-46b1-810b-4567363be4eb"). InnerVolumeSpecName "kube-api-access-dwd2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:56.541838 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.541604 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-envoy" pod "94e4cea7-0d7d-46b1-810b-4567363be4eb" (UID: "94e4cea7-0d7d-46b1-810b-4567363be4eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.541838 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.541677 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-token" pod "94e4cea7-0d7d-46b1-810b-4567363be4eb" (UID: "94e4cea7-0d7d-46b1-810b-4567363be4eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:56.637166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.637131 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.637166 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.637165 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.637383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.637179 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwd2r\" (UniqueName: \"kubernetes.io/projected/94e4cea7-0d7d-46b1-810b-4567363be4eb-kube-api-access-dwd2r\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.637383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.637192 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.637383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.637204 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.637383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.637214 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.637383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.637226 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e4cea7-0d7d-46b1-810b-4567363be4eb-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.637383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.637241 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e4cea7-0d7d-46b1-810b-4567363be4eb-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.637383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.637252 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e4cea7-0d7d-46b1-810b-4567363be4eb-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.650883 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.650814 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:45:56.708224 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.708186 5998 generic.go:356] "Generic (PLEG): container finished" podID="13bc0eed-818a-4ef8-bc7a-6d9a2aed513f" containerID="1c6c1bd95fe09c8b35543adea05056e16b5b0c674a9543da57ac934e02a2a234" exitCode=0 Oct 30 02:45:56.708224 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.708230 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" Oct 30 02:45:56.708539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.708281 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" event={"ID":"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f","Type":"ContainerDied","Data":"1c6c1bd95fe09c8b35543adea05056e16b5b0c674a9543da57ac934e02a2a234"} Oct 30 02:45:56.708539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.708336 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r" event={"ID":"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f","Type":"ContainerDied","Data":"0740ffd990d1007a9915a379f8227cfd2866a32b5b42378db966e3ebcacf75c9"} Oct 30 02:45:56.708539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.708371 5998 scope.go:122] "RemoveContainer" containerID="1c6c1bd95fe09c8b35543adea05056e16b5b0c674a9543da57ac934e02a2a234" Oct 30 02:45:56.710145 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.710116 5998 generic.go:356] "Generic (PLEG): container finished" podID="c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" containerID="0470cb631abc84e26e55ca4d39d66bc5d8c35568914bb2540fdb1346db1d4588" exitCode=0 Oct 30 02:45:56.710259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.710159 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" Oct 30 02:45:56.710259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.710213 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" event={"ID":"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b","Type":"ContainerDied","Data":"0470cb631abc84e26e55ca4d39d66bc5d8c35568914bb2540fdb1346db1d4588"} Oct 30 02:45:56.710392 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.710270 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs" event={"ID":"c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b","Type":"ContainerDied","Data":"2b792f944f0b79a1b4d5c2767c1d16a858ec4dbd36b42542fea22b56ae484f41"} Oct 30 02:45:56.711981 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.711954 5998 generic.go:356] "Generic (PLEG): container finished" podID="42fa7bf8-25e0-4bd8-b751-243b12d986b3" containerID="4b7c66f045a1d2292dd404ae604ee47c603d9e0acd3a591c7aa5f3243f2a2dd9" exitCode=0 Oct 30 02:45:56.712131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.712057 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" Oct 30 02:45:56.712207 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.712144 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" event={"ID":"42fa7bf8-25e0-4bd8-b751-243b12d986b3","Type":"ContainerDied","Data":"4b7c66f045a1d2292dd404ae604ee47c603d9e0acd3a591c7aa5f3243f2a2dd9"} Oct 30 02:45:56.712738 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.712669 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn" event={"ID":"42fa7bf8-25e0-4bd8-b751-243b12d986b3","Type":"ContainerDied","Data":"4489792261eb1a3e27a8ca3c5e5fbccdf7c9e375ad9ace63c3243fbc66a342ae"} Oct 30 02:45:56.714252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.714234 5998 generic.go:356] "Generic (PLEG): container finished" podID="acb7072a-7343-4122-9472-93c1a658cb02" containerID="8b0c1bb4a6b32938478201745bc90f2d0ffbcfe624caf4f6a1be410f982de16b" exitCode=0 Oct 30 02:45:56.714369 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.714295 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" Oct 30 02:45:56.714369 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.714322 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" event={"ID":"acb7072a-7343-4122-9472-93c1a658cb02","Type":"ContainerDied","Data":"8b0c1bb4a6b32938478201745bc90f2d0ffbcfe624caf4f6a1be410f982de16b"} Oct 30 02:45:56.714482 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.714371 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m" event={"ID":"acb7072a-7343-4122-9472-93c1a658cb02","Type":"ContainerDied","Data":"006656527df8a1bb3121f86fb68b46b40ed383e2e6a2c3561aa0c1c7b963d27d"} Oct 30 02:45:56.715992 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.715968 5998 generic.go:356] "Generic (PLEG): container finished" podID="94e4cea7-0d7d-46b1-810b-4567363be4eb" containerID="dd9f169eaba798b00bc61f97c0fdb48011ba289ff77db7b62c63c688eb87fcc5" exitCode=0 Oct 30 02:45:56.716112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.716006 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" Oct 30 02:45:56.716112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.716012 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" event={"ID":"94e4cea7-0d7d-46b1-810b-4567363be4eb","Type":"ContainerDied","Data":"dd9f169eaba798b00bc61f97c0fdb48011ba289ff77db7b62c63c688eb87fcc5"} Oct 30 02:45:56.716112 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.716053 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm" event={"ID":"94e4cea7-0d7d-46b1-810b-4567363be4eb","Type":"ContainerDied","Data":"97eb5b310e2bf74c004f9f1e134c9803303d63d5820e6bec4de14b090eea1c38"} Oct 30 02:45:56.723834 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.723658 5998 scope.go:122] "RemoveContainer" containerID="1c6c1bd95fe09c8b35543adea05056e16b5b0c674a9543da57ac934e02a2a234" Oct 30 02:45:56.724006 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:56.723985 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6c1bd95fe09c8b35543adea05056e16b5b0c674a9543da57ac934e02a2a234\": container with ID starting with 1c6c1bd95fe09c8b35543adea05056e16b5b0c674a9543da57ac934e02a2a234 not found: ID does not exist" containerID="1c6c1bd95fe09c8b35543adea05056e16b5b0c674a9543da57ac934e02a2a234" Oct 30 02:45:56.724069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.724016 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6c1bd95fe09c8b35543adea05056e16b5b0c674a9543da57ac934e02a2a234"} err="failed to get container status \"1c6c1bd95fe09c8b35543adea05056e16b5b0c674a9543da57ac934e02a2a234\": rpc error: code = NotFound desc = could not find container \"1c6c1bd95fe09c8b35543adea05056e16b5b0c674a9543da57ac934e02a2a234\": container with ID starting with 1c6c1bd95fe09c8b35543adea05056e16b5b0c674a9543da57ac934e02a2a234 not found: ID does not exist" Oct 30 02:45:56.724069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.724034 5998 scope.go:122] "RemoveContainer" containerID="0470cb631abc84e26e55ca4d39d66bc5d8c35568914bb2540fdb1346db1d4588" Oct 30 02:45:56.735538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.735512 5998 scope.go:122] "RemoveContainer" containerID="0470cb631abc84e26e55ca4d39d66bc5d8c35568914bb2540fdb1346db1d4588" Oct 30 02:45:56.735836 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:56.735812 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0470cb631abc84e26e55ca4d39d66bc5d8c35568914bb2540fdb1346db1d4588\": container with ID starting with 0470cb631abc84e26e55ca4d39d66bc5d8c35568914bb2540fdb1346db1d4588 not found: ID does not exist" containerID="0470cb631abc84e26e55ca4d39d66bc5d8c35568914bb2540fdb1346db1d4588" Oct 30 02:45:56.735928 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.735843 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0470cb631abc84e26e55ca4d39d66bc5d8c35568914bb2540fdb1346db1d4588"} err="failed to get container status \"0470cb631abc84e26e55ca4d39d66bc5d8c35568914bb2540fdb1346db1d4588\": rpc error: code = NotFound desc = could not find container \"0470cb631abc84e26e55ca4d39d66bc5d8c35568914bb2540fdb1346db1d4588\": container with ID starting with 0470cb631abc84e26e55ca4d39d66bc5d8c35568914bb2540fdb1346db1d4588 not found: ID does not exist" Oct 30 02:45:56.735928 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.735870 5998 scope.go:122] "RemoveContainer" containerID="4b7c66f045a1d2292dd404ae604ee47c603d9e0acd3a591c7aa5f3243f2a2dd9" Oct 30 02:45:56.738267 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.738243 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-tmp-dir\") pod \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " Oct 30 02:45:56.738377 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.738303 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-model-cache\") pod \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " Oct 30 02:45:56.738494 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.738477 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-home\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-home\") pod \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " Oct 30 02:45:56.738607 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.738559 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-tmp-dir" pod "13bc0eed-818a-4ef8-bc7a-6d9a2aed513f" (UID: "13bc0eed-818a-4ef8-bc7a-6d9a2aed513f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.738738 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.738635 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-model-cache" pod "13bc0eed-818a-4ef8-bc7a-6d9a2aed513f" (UID: "13bc0eed-818a-4ef8-bc7a-6d9a2aed513f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.738804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.738769 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-home" pod "13bc0eed-818a-4ef8-bc7a-6d9a2aed513f" (UID: "13bc0eed-818a-4ef8-bc7a-6d9a2aed513f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.738804 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.738572 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-kube-api-access-jnzh2\" (UniqueName: \"kubernetes.io/projected/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-kube-api-access-jnzh2\") pod \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " Oct 30 02:45:56.738915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.738831 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-tls-certs\" (UniqueName: \"kubernetes.io/secret/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-tls-certs\") pod \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " Oct 30 02:45:56.738915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.738874 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-dshm\") pod \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\" (UID: \"13bc0eed-818a-4ef8-bc7a-6d9a2aed513f\") " Oct 30 02:45:56.739452 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.739427 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.739551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.739456 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.739551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.739470 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.740683 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.740655 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-kube-api-access-jnzh2" pod "13bc0eed-818a-4ef8-bc7a-6d9a2aed513f" (UID: "13bc0eed-818a-4ef8-bc7a-6d9a2aed513f"). InnerVolumeSpecName "kube-api-access-jnzh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:45:56.740861 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.740844 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-dshm" pod "13bc0eed-818a-4ef8-bc7a-6d9a2aed513f" (UID: "13bc0eed-818a-4ef8-bc7a-6d9a2aed513f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:45:56.740948 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.740929 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-tls-certs" pod "13bc0eed-818a-4ef8-bc7a-6d9a2aed513f" (UID: "13bc0eed-818a-4ef8-bc7a-6d9a2aed513f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:45:56.757642 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.757615 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn"] Oct 30 02:45:56.761835 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.761801 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-mzxsn"] Oct 30 02:45:56.767161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.767138 5998 scope.go:122] "RemoveContainer" containerID="4b7c66f045a1d2292dd404ae604ee47c603d9e0acd3a591c7aa5f3243f2a2dd9" Oct 30 02:45:56.767546 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:56.767518 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b7c66f045a1d2292dd404ae604ee47c603d9e0acd3a591c7aa5f3243f2a2dd9\": container with ID starting with 4b7c66f045a1d2292dd404ae604ee47c603d9e0acd3a591c7aa5f3243f2a2dd9 not found: ID does not exist" containerID="4b7c66f045a1d2292dd404ae604ee47c603d9e0acd3a591c7aa5f3243f2a2dd9" Oct 30 02:45:56.767605 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.767554 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b7c66f045a1d2292dd404ae604ee47c603d9e0acd3a591c7aa5f3243f2a2dd9"} err="failed to get container status \"4b7c66f045a1d2292dd404ae604ee47c603d9e0acd3a591c7aa5f3243f2a2dd9\": rpc error: code = NotFound desc = could not find container \"4b7c66f045a1d2292dd404ae604ee47c603d9e0acd3a591c7aa5f3243f2a2dd9\": container with ID starting with 4b7c66f045a1d2292dd404ae604ee47c603d9e0acd3a591c7aa5f3243f2a2dd9 not found: ID does not exist" Oct 30 02:45:56.767605 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.767573 5998 scope.go:122] "RemoveContainer" containerID="8b0c1bb4a6b32938478201745bc90f2d0ffbcfe624caf4f6a1be410f982de16b" Oct 30 02:45:56.777696 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.777677 5998 scope.go:122] "RemoveContainer" containerID="8b0c1bb4a6b32938478201745bc90f2d0ffbcfe624caf4f6a1be410f982de16b" Oct 30 02:45:56.777986 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:56.777966 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0c1bb4a6b32938478201745bc90f2d0ffbcfe624caf4f6a1be410f982de16b\": container with ID starting with 8b0c1bb4a6b32938478201745bc90f2d0ffbcfe624caf4f6a1be410f982de16b not found: ID does not exist" containerID="8b0c1bb4a6b32938478201745bc90f2d0ffbcfe624caf4f6a1be410f982de16b" Oct 30 02:45:56.778071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.777996 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0c1bb4a6b32938478201745bc90f2d0ffbcfe624caf4f6a1be410f982de16b"} err="failed to get container status \"8b0c1bb4a6b32938478201745bc90f2d0ffbcfe624caf4f6a1be410f982de16b\": rpc error: code = NotFound desc = could not find container \"8b0c1bb4a6b32938478201745bc90f2d0ffbcfe624caf4f6a1be410f982de16b\": container with ID starting with 8b0c1bb4a6b32938478201745bc90f2d0ffbcfe624caf4f6a1be410f982de16b not found: ID does not exist" Oct 30 02:45:56.778071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.778018 5998 scope.go:122] "RemoveContainer" containerID="dd9f169eaba798b00bc61f97c0fdb48011ba289ff77db7b62c63c688eb87fcc5" Oct 30 02:45:56.789286 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.789262 5998 scope.go:122] "RemoveContainer" containerID="dd9f169eaba798b00bc61f97c0fdb48011ba289ff77db7b62c63c688eb87fcc5" Oct 30 02:45:56.789815 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:45:56.789791 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd9f169eaba798b00bc61f97c0fdb48011ba289ff77db7b62c63c688eb87fcc5\": container with ID starting with dd9f169eaba798b00bc61f97c0fdb48011ba289ff77db7b62c63c688eb87fcc5 not found: ID does not exist" containerID="dd9f169eaba798b00bc61f97c0fdb48011ba289ff77db7b62c63c688eb87fcc5" Oct 30 02:45:56.789902 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.789827 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd9f169eaba798b00bc61f97c0fdb48011ba289ff77db7b62c63c688eb87fcc5"} err="failed to get container status \"dd9f169eaba798b00bc61f97c0fdb48011ba289ff77db7b62c63c688eb87fcc5\": rpc error: code = NotFound desc = could not find container \"dd9f169eaba798b00bc61f97c0fdb48011ba289ff77db7b62c63c688eb87fcc5\": container with ID starting with dd9f169eaba798b00bc61f97c0fdb48011ba289ff77db7b62c63c688eb87fcc5 not found: ID does not exist" Oct 30 02:45:56.791711 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.791682 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m"] Oct 30 02:45:56.795449 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.795422 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-m9q2m"] Oct 30 02:45:56.840553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.840512 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnzh2\" (UniqueName: \"kubernetes.io/projected/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-kube-api-access-jnzh2\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.840553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.840548 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.840769 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.840564 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:45:56.841636 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.841605 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs"] Oct 30 02:45:56.846964 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.846933 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-2qvfs"] Oct 30 02:45:56.876215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.876175 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm"] Oct 30 02:45:56.882130 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:56.882082 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-bwczm"] Oct 30 02:45:57.051768 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:57.051733 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r"] Oct 30 02:45:57.055176 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:57.055143 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-2819d2/sub-test-model-2819d2-kserve-7fc9bc689c-bnc4r"] Oct 30 02:45:57.344485 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:57.344368 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:57.974019 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:57.973984 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:45:58.539619 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:58.539580 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13bc0eed-818a-4ef8-bc7a-6d9a2aed513f" path="/var/lib/kubelet/pods/13bc0eed-818a-4ef8-bc7a-6d9a2aed513f/volumes" Oct 30 02:45:58.540270 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:58.540232 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42fa7bf8-25e0-4bd8-b751-243b12d986b3" path="/var/lib/kubelet/pods/42fa7bf8-25e0-4bd8-b751-243b12d986b3/volumes" Oct 30 02:45:58.540747 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:58.540728 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e4cea7-0d7d-46b1-810b-4567363be4eb" path="/var/lib/kubelet/pods/94e4cea7-0d7d-46b1-810b-4567363be4eb/volumes" Oct 30 02:45:58.541487 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:58.541471 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acb7072a-7343-4122-9472-93c1a658cb02" path="/var/lib/kubelet/pods/acb7072a-7343-4122-9472-93c1a658cb02/volumes" Oct 30 02:45:58.541902 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:58.541889 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" path="/var/lib/kubelet/pods/c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b/volumes" Oct 30 02:45:59.476022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:45:59.475981 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:01.206199 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:01.188084 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:02.687444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:02.687405 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:02.829810 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:02.829766 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:03.175413 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:03.175378 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:03.366608 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:03.366576 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:03.724114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:03.723107 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj"] Oct 30 02:46:03.724114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:03.723385 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" podUID="af31090b-f48b-4675-9add-665d0f5d42af" containerName="istio-proxy" containerID="cri-o://b34f8961d59280e60a050eb9af1aef1a2ed40e9c253ef77f7b4519a0d23dd5eb" gracePeriod=30 Oct 30 02:46:03.752969 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:03.752922 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:04.484496 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:04.484458 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:04.600860 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:04.600820 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:04.717444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:04.717411 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:06.284504 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:06.284466 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:06.579060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:06.578965 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:46:06.579060 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:06.579024 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" podUID="af31090b-f48b-4675-9add-665d0f5d42af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:46:06.779553 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:06.779513 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:46:07.661275 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:07.661238 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:08.357499 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:08.357456 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:09.018155 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.018052 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:46:09.112684 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.112648 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-istio-envoy\") pod \"af31090b-f48b-4675-9add-665d0f5d42af\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " Oct 30 02:46:09.112955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.112722 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-workload-socket\") pod \"af31090b-f48b-4675-9add-665d0f5d42af\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " Oct 30 02:46:09.112955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.112770 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-istio-data\") pod \"af31090b-f48b-4675-9add-665d0f5d42af\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " Oct 30 02:46:09.112955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.112799 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-credential-socket\") pod \"af31090b-f48b-4675-9add-665d0f5d42af\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " Oct 30 02:46:09.112955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.112860 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-workload-certs\") pod \"af31090b-f48b-4675-9add-665d0f5d42af\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " Oct 30 02:46:09.112955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.112918 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af31090b-f48b-4675-9add-665d0f5d42af-kube-api-access-87fr9\" (UniqueName: \"kubernetes.io/projected/af31090b-f48b-4675-9add-665d0f5d42af-kube-api-access-87fr9\") pod \"af31090b-f48b-4675-9add-665d0f5d42af\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " Oct 30 02:46:09.113279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.112975 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af31090b-f48b-4675-9add-665d0f5d42af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af31090b-f48b-4675-9add-665d0f5d42af-istio-podinfo\") pod \"af31090b-f48b-4675-9add-665d0f5d42af\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " Oct 30 02:46:09.113279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.113016 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af31090b-f48b-4675-9add-665d0f5d42af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af31090b-f48b-4675-9add-665d0f5d42af-istiod-ca-cert\") pod \"af31090b-f48b-4675-9add-665d0f5d42af\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " Oct 30 02:46:09.113279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.113050 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af31090b-f48b-4675-9add-665d0f5d42af-istio-token\" (UniqueName: \"kubernetes.io/projected/af31090b-f48b-4675-9add-665d0f5d42af-istio-token\") pod \"af31090b-f48b-4675-9add-665d0f5d42af\" (UID: \"af31090b-f48b-4675-9add-665d0f5d42af\") " Oct 30 02:46:09.113279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.113169 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-workload-socket" pod "af31090b-f48b-4675-9add-665d0f5d42af" (UID: "af31090b-f48b-4675-9add-665d0f5d42af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:09.113514 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.113447 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-workload-certs" pod "af31090b-f48b-4675-9add-665d0f5d42af" (UID: "af31090b-f48b-4675-9add-665d0f5d42af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:09.113582 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.113536 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-credential-socket" pod "af31090b-f48b-4675-9add-665d0f5d42af" (UID: "af31090b-f48b-4675-9add-665d0f5d42af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:09.113793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.113766 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af31090b-f48b-4675-9add-665d0f5d42af-istiod-ca-cert" pod "af31090b-f48b-4675-9add-665d0f5d42af" (UID: "af31090b-f48b-4675-9add-665d0f5d42af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:09.113793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.113778 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-istio-data" pod "af31090b-f48b-4675-9add-665d0f5d42af" (UID: "af31090b-f48b-4675-9add-665d0f5d42af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:09.114252 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.114233 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:09.114412 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.114256 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:09.114412 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.114270 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:09.114412 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.114283 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af31090b-f48b-4675-9add-665d0f5d42af-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:09.114412 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.114296 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:09.115891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.115864 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-istio-envoy" pod "af31090b-f48b-4675-9add-665d0f5d42af" (UID: "af31090b-f48b-4675-9add-665d0f5d42af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:09.116076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.116053 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af31090b-f48b-4675-9add-665d0f5d42af-kube-api-access-87fr9" pod "af31090b-f48b-4675-9add-665d0f5d42af" (UID: "af31090b-f48b-4675-9add-665d0f5d42af"). InnerVolumeSpecName "kube-api-access-87fr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:09.116412 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.116386 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af31090b-f48b-4675-9add-665d0f5d42af-istio-token" pod "af31090b-f48b-4675-9add-665d0f5d42af" (UID: "af31090b-f48b-4675-9add-665d0f5d42af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:09.116624 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.116600 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af31090b-f48b-4675-9add-665d0f5d42af-istio-podinfo" pod "af31090b-f48b-4675-9add-665d0f5d42af" (UID: "af31090b-f48b-4675-9add-665d0f5d42af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:46:09.215294 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.215256 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87fr9\" (UniqueName: \"kubernetes.io/projected/af31090b-f48b-4675-9add-665d0f5d42af-kube-api-access-87fr9\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:09.215294 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.215289 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af31090b-f48b-4675-9add-665d0f5d42af-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:09.215294 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.215299 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af31090b-f48b-4675-9add-665d0f5d42af-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:09.215573 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.215307 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af31090b-f48b-4675-9add-665d0f5d42af-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:09.586049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.586001 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" podUID="5f859aa2-e664-4e9b-b8d7-474991da223d" containerName="main" containerID="cri-o://83836b039d3c67e4479858d1f1fc5ed2cda2052ab5e91adcfffe076d7aec8da6" gracePeriod=45 Oct 30 02:46:09.831306 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.831271 5998 generic.go:356] "Generic (PLEG): container finished" podID="af31090b-f48b-4675-9add-665d0f5d42af" containerID="b34f8961d59280e60a050eb9af1aef1a2ed40e9c253ef77f7b4519a0d23dd5eb" exitCode=0 Oct 30 02:46:09.831584 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.831312 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" event={"ID":"af31090b-f48b-4675-9add-665d0f5d42af","Type":"ContainerDied","Data":"b34f8961d59280e60a050eb9af1aef1a2ed40e9c253ef77f7b4519a0d23dd5eb"} Oct 30 02:46:09.831707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.831338 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" Oct 30 02:46:09.831707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.831627 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj" event={"ID":"af31090b-f48b-4675-9add-665d0f5d42af","Type":"ContainerDied","Data":"26c5f0ee224c1d84f1043fd456925a40caf831d1394dd78a4eb726ec57e5a168"} Oct 30 02:46:09.831707 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.831663 5998 scope.go:122] "RemoveContainer" containerID="b34f8961d59280e60a050eb9af1aef1a2ed40e9c253ef77f7b4519a0d23dd5eb" Oct 30 02:46:09.835073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.835045 5998 generic.go:356] "Generic (PLEG): container finished" podID="5f859aa2-e664-4e9b-b8d7-474991da223d" containerID="83836b039d3c67e4479858d1f1fc5ed2cda2052ab5e91adcfffe076d7aec8da6" exitCode=0 Oct 30 02:46:09.835262 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.835129 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" event={"ID":"5f859aa2-e664-4e9b-b8d7-474991da223d","Type":"ContainerDied","Data":"83836b039d3c67e4479858d1f1fc5ed2cda2052ab5e91adcfffe076d7aec8da6"} Oct 30 02:46:09.844305 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.844282 5998 scope.go:122] "RemoveContainer" containerID="e9dc92290e7b950f887056916718f686a3923657c26628f386896b13c4723dec" Oct 30 02:46:09.865022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.864825 5998 scope.go:122] "RemoveContainer" containerID="b34f8961d59280e60a050eb9af1aef1a2ed40e9c253ef77f7b4519a0d23dd5eb" Oct 30 02:46:09.865184 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:46:09.865162 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34f8961d59280e60a050eb9af1aef1a2ed40e9c253ef77f7b4519a0d23dd5eb\": container with ID starting with b34f8961d59280e60a050eb9af1aef1a2ed40e9c253ef77f7b4519a0d23dd5eb not found: ID does not exist" containerID="b34f8961d59280e60a050eb9af1aef1a2ed40e9c253ef77f7b4519a0d23dd5eb" Oct 30 02:46:09.865246 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.865193 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34f8961d59280e60a050eb9af1aef1a2ed40e9c253ef77f7b4519a0d23dd5eb"} err="failed to get container status \"b34f8961d59280e60a050eb9af1aef1a2ed40e9c253ef77f7b4519a0d23dd5eb\": rpc error: code = NotFound desc = could not find container \"b34f8961d59280e60a050eb9af1aef1a2ed40e9c253ef77f7b4519a0d23dd5eb\": container with ID starting with b34f8961d59280e60a050eb9af1aef1a2ed40e9c253ef77f7b4519a0d23dd5eb not found: ID does not exist" Oct 30 02:46:09.865246 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.865209 5998 scope.go:122] "RemoveContainer" containerID="e9dc92290e7b950f887056916718f686a3923657c26628f386896b13c4723dec" Oct 30 02:46:09.865477 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:46:09.865456 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9dc92290e7b950f887056916718f686a3923657c26628f386896b13c4723dec\": container with ID starting with e9dc92290e7b950f887056916718f686a3923657c26628f386896b13c4723dec not found: ID does not exist" containerID="e9dc92290e7b950f887056916718f686a3923657c26628f386896b13c4723dec" Oct 30 02:46:09.865543 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.865488 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9dc92290e7b950f887056916718f686a3923657c26628f386896b13c4723dec"} err="failed to get container status \"e9dc92290e7b950f887056916718f686a3923657c26628f386896b13c4723dec\": rpc error: code = NotFound desc = could not find container \"e9dc92290e7b950f887056916718f686a3923657c26628f386896b13c4723dec\": container with ID starting with e9dc92290e7b950f887056916718f686a3923657c26628f386896b13c4723dec not found: ID does not exist" Oct 30 02:46:09.868611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.868587 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:46:09.886790 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.886762 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:09.896572 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.896541 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj"] Oct 30 02:46:09.907718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:09.907676 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-6b8bfa-openshift-default-7fc555776f-p8scj"] Oct 30 02:46:10.023346 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.023315 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-dshm\") pod \"5f859aa2-e664-4e9b-b8d7-474991da223d\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " Oct 30 02:46:10.023852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.023359 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-tmp-dir\") pod \"5f859aa2-e664-4e9b-b8d7-474991da223d\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " Oct 30 02:46:10.023852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.023386 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f859aa2-e664-4e9b-b8d7-474991da223d-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f859aa2-e664-4e9b-b8d7-474991da223d-tls-certs\") pod \"5f859aa2-e664-4e9b-b8d7-474991da223d\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " Oct 30 02:46:10.023852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.023489 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f859aa2-e664-4e9b-b8d7-474991da223d-kube-api-access-k7lkf\" (UniqueName: \"kubernetes.io/projected/5f859aa2-e664-4e9b-b8d7-474991da223d-kube-api-access-k7lkf\") pod \"5f859aa2-e664-4e9b-b8d7-474991da223d\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " Oct 30 02:46:10.023852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.023534 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-home\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-home\") pod \"5f859aa2-e664-4e9b-b8d7-474991da223d\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " Oct 30 02:46:10.023852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.023589 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-model-cache\") pod \"5f859aa2-e664-4e9b-b8d7-474991da223d\" (UID: \"5f859aa2-e664-4e9b-b8d7-474991da223d\") " Oct 30 02:46:10.023852 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.023658 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-tmp-dir" pod "5f859aa2-e664-4e9b-b8d7-474991da223d" (UID: "5f859aa2-e664-4e9b-b8d7-474991da223d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:10.024215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.023858 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-home" pod "5f859aa2-e664-4e9b-b8d7-474991da223d" (UID: "5f859aa2-e664-4e9b-b8d7-474991da223d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:10.024215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.023944 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-model-cache" pod "5f859aa2-e664-4e9b-b8d7-474991da223d" (UID: "5f859aa2-e664-4e9b-b8d7-474991da223d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:10.024215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.024135 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:10.024215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.024172 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:10.024215 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.024189 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:10.025871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.025851 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f859aa2-e664-4e9b-b8d7-474991da223d-tls-certs" pod "5f859aa2-e664-4e9b-b8d7-474991da223d" (UID: "5f859aa2-e664-4e9b-b8d7-474991da223d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:46:10.026010 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.025993 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-dshm" pod "5f859aa2-e664-4e9b-b8d7-474991da223d" (UID: "5f859aa2-e664-4e9b-b8d7-474991da223d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:10.026083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.026058 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f859aa2-e664-4e9b-b8d7-474991da223d-kube-api-access-k7lkf" pod "5f859aa2-e664-4e9b-b8d7-474991da223d" (UID: "5f859aa2-e664-4e9b-b8d7-474991da223d"). InnerVolumeSpecName "kube-api-access-k7lkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:10.124229 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.124139 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:10.124755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.124736 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f859aa2-e664-4e9b-b8d7-474991da223d-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:10.124809 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.124759 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f859aa2-e664-4e9b-b8d7-474991da223d-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:10.124809 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.124769 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7lkf\" (UniqueName: \"kubernetes.io/projected/5f859aa2-e664-4e9b-b8d7-474991da223d-kube-api-access-k7lkf\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:10.538448 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.538416 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af31090b-f48b-4675-9add-665d0f5d42af" path="/var/lib/kubelet/pods/af31090b-f48b-4675-9add-665d0f5d42af/volumes" Oct 30 02:46:10.845652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.845561 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" Oct 30 02:46:10.845652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.845596 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf" event={"ID":"5f859aa2-e664-4e9b-b8d7-474991da223d","Type":"ContainerDied","Data":"0c368c63ff914345a4ed1c9251564dc361ca62a64ebb086056d94907f2373532"} Oct 30 02:46:10.845652 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.845648 5998 scope.go:122] "RemoveContainer" containerID="83836b039d3c67e4479858d1f1fc5ed2cda2052ab5e91adcfffe076d7aec8da6" Oct 30 02:46:10.893914 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.893877 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf"] Oct 30 02:46:10.904217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:10.904185 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9a1cd0/sub-test-model-9a1cd0-kserve-74fccb865c-tgqkf"] Oct 30 02:46:11.577288 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:11.577245 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:11.763253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:11.763160 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:11.879372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:11.879278 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:12.301677 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:12.301646 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:12.519867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:12.519801 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:12.540154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:12.540114 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f859aa2-e664-4e9b-b8d7-474991da223d" path="/var/lib/kubelet/pods/5f859aa2-e664-4e9b-b8d7-474991da223d/volumes" Oct 30 02:46:12.630772 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:12.630687 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:12.756365 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:12.756319 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:12.890741 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:12.890642 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg"] Oct 30 02:46:12.890931 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:12.890868 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" podUID="b13ac864-0406-4db5-b911-2851f86df2b6" containerName="istio-proxy" containerID="cri-o://d31176aa8e60ad566c31291b46dd242581dbf1080e275625f91f4501017a382c" gracePeriod=30 Oct 30 02:46:12.939488 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:12.939444 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:13.508418 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:13.508372 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:13.789723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:13.789632 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:15.252506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:15.252467 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:16.730751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:16.730715 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:18.157540 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.157472 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:46:18.219713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.219682 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-workload-certs\") pod \"b13ac864-0406-4db5-b911-2851f86df2b6\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " Oct 30 02:46:18.219951 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.219748 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b13ac864-0406-4db5-b911-2851f86df2b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b13ac864-0406-4db5-b911-2851f86df2b6-istio-podinfo\") pod \"b13ac864-0406-4db5-b911-2851f86df2b6\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " Oct 30 02:46:18.219951 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.219793 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-istio-data\") pod \"b13ac864-0406-4db5-b911-2851f86df2b6\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " Oct 30 02:46:18.219951 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.219864 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b13ac864-0406-4db5-b911-2851f86df2b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b13ac864-0406-4db5-b911-2851f86df2b6-istiod-ca-cert\") pod \"b13ac864-0406-4db5-b911-2851f86df2b6\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " Oct 30 02:46:18.219951 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.219931 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-credential-socket\") pod \"b13ac864-0406-4db5-b911-2851f86df2b6\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " Oct 30 02:46:18.220318 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.220050 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b13ac864-0406-4db5-b911-2851f86df2b6-kube-api-access-dhvp8\" (UniqueName: \"kubernetes.io/projected/b13ac864-0406-4db5-b911-2851f86df2b6-kube-api-access-dhvp8\") pod \"b13ac864-0406-4db5-b911-2851f86df2b6\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " Oct 30 02:46:18.220318 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.220142 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b13ac864-0406-4db5-b911-2851f86df2b6-istio-token\" (UniqueName: \"kubernetes.io/projected/b13ac864-0406-4db5-b911-2851f86df2b6-istio-token\") pod \"b13ac864-0406-4db5-b911-2851f86df2b6\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " Oct 30 02:46:18.220318 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.220187 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-istio-envoy\") pod \"b13ac864-0406-4db5-b911-2851f86df2b6\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " Oct 30 02:46:18.220318 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.220235 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-workload-socket\") pod \"b13ac864-0406-4db5-b911-2851f86df2b6\" (UID: \"b13ac864-0406-4db5-b911-2851f86df2b6\") " Oct 30 02:46:18.221328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.221300 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-credential-socket" pod "b13ac864-0406-4db5-b911-2851f86df2b6" (UID: "b13ac864-0406-4db5-b911-2851f86df2b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:18.222110 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.221891 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-workload-certs" pod "b13ac864-0406-4db5-b911-2851f86df2b6" (UID: "b13ac864-0406-4db5-b911-2851f86df2b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:18.222433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.222364 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-workload-socket" pod "b13ac864-0406-4db5-b911-2851f86df2b6" (UID: "b13ac864-0406-4db5-b911-2851f86df2b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:18.222560 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.222428 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b13ac864-0406-4db5-b911-2851f86df2b6-istiod-ca-cert" pod "b13ac864-0406-4db5-b911-2851f86df2b6" (UID: "b13ac864-0406-4db5-b911-2851f86df2b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:18.222560 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.222527 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-istio-data" pod "b13ac864-0406-4db5-b911-2851f86df2b6" (UID: "b13ac864-0406-4db5-b911-2851f86df2b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:18.223725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.223083 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b13ac864-0406-4db5-b911-2851f86df2b6-istio-podinfo" pod "b13ac864-0406-4db5-b911-2851f86df2b6" (UID: "b13ac864-0406-4db5-b911-2851f86df2b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:46:18.223997 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.223954 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-istio-envoy" pod "b13ac864-0406-4db5-b911-2851f86df2b6" (UID: "b13ac864-0406-4db5-b911-2851f86df2b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:18.224152 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.224060 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b13ac864-0406-4db5-b911-2851f86df2b6-kube-api-access-dhvp8" pod "b13ac864-0406-4db5-b911-2851f86df2b6" (UID: "b13ac864-0406-4db5-b911-2851f86df2b6"). InnerVolumeSpecName "kube-api-access-dhvp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:18.224564 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.224540 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b13ac864-0406-4db5-b911-2851f86df2b6-istio-token" pod "b13ac864-0406-4db5-b911-2851f86df2b6" (UID: "b13ac864-0406-4db5-b911-2851f86df2b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:18.284538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.284501 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:18.322098 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.322048 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhvp8\" (UniqueName: \"kubernetes.io/projected/b13ac864-0406-4db5-b911-2851f86df2b6-kube-api-access-dhvp8\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:18.322279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.322084 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b13ac864-0406-4db5-b911-2851f86df2b6-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:18.322279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.322121 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:18.322279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.322129 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:18.322279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.322138 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:18.322279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.322146 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b13ac864-0406-4db5-b911-2851f86df2b6-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:18.322279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.322154 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:18.322279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.322161 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b13ac864-0406-4db5-b911-2851f86df2b6-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:18.322279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.322169 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13ac864-0406-4db5-b911-2851f86df2b6-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:18.917251 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.917213 5998 generic.go:356] "Generic (PLEG): container finished" podID="b13ac864-0406-4db5-b911-2851f86df2b6" containerID="d31176aa8e60ad566c31291b46dd242581dbf1080e275625f91f4501017a382c" exitCode=0 Oct 30 02:46:18.917410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.917272 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" Oct 30 02:46:18.917410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.917298 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" event={"ID":"b13ac864-0406-4db5-b911-2851f86df2b6","Type":"ContainerDied","Data":"d31176aa8e60ad566c31291b46dd242581dbf1080e275625f91f4501017a382c"} Oct 30 02:46:18.917410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.917360 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg" event={"ID":"b13ac864-0406-4db5-b911-2851f86df2b6","Type":"ContainerDied","Data":"b006f99212ab45a0e8ceaeb34bcf93eee2fc055fc412385f0ae984873151dea0"} Oct 30 02:46:18.917410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.917392 5998 scope.go:122] "RemoveContainer" containerID="d31176aa8e60ad566c31291b46dd242581dbf1080e275625f91f4501017a382c" Oct 30 02:46:18.927811 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.927790 5998 scope.go:122] "RemoveContainer" containerID="81cb1590c843dc22421c710b134b517b7ca49efb367815ff446e77a825311f55" Oct 30 02:46:18.937379 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.937356 5998 scope.go:122] "RemoveContainer" containerID="d31176aa8e60ad566c31291b46dd242581dbf1080e275625f91f4501017a382c" Oct 30 02:46:18.937686 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:46:18.937663 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d31176aa8e60ad566c31291b46dd242581dbf1080e275625f91f4501017a382c\": container with ID starting with d31176aa8e60ad566c31291b46dd242581dbf1080e275625f91f4501017a382c not found: ID does not exist" containerID="d31176aa8e60ad566c31291b46dd242581dbf1080e275625f91f4501017a382c" Oct 30 02:46:18.937768 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.937700 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d31176aa8e60ad566c31291b46dd242581dbf1080e275625f91f4501017a382c"} err="failed to get container status \"d31176aa8e60ad566c31291b46dd242581dbf1080e275625f91f4501017a382c\": rpc error: code = NotFound desc = could not find container \"d31176aa8e60ad566c31291b46dd242581dbf1080e275625f91f4501017a382c\": container with ID starting with d31176aa8e60ad566c31291b46dd242581dbf1080e275625f91f4501017a382c not found: ID does not exist" Oct 30 02:46:18.937768 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.937726 5998 scope.go:122] "RemoveContainer" containerID="81cb1590c843dc22421c710b134b517b7ca49efb367815ff446e77a825311f55" Oct 30 02:46:18.938009 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:46:18.937989 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81cb1590c843dc22421c710b134b517b7ca49efb367815ff446e77a825311f55\": container with ID starting with 81cb1590c843dc22421c710b134b517b7ca49efb367815ff446e77a825311f55 not found: ID does not exist" containerID="81cb1590c843dc22421c710b134b517b7ca49efb367815ff446e77a825311f55" Oct 30 02:46:18.938050 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.938016 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81cb1590c843dc22421c710b134b517b7ca49efb367815ff446e77a825311f55"} err="failed to get container status \"81cb1590c843dc22421c710b134b517b7ca49efb367815ff446e77a825311f55\": rpc error: code = NotFound desc = could not find container \"81cb1590c843dc22421c710b134b517b7ca49efb367815ff446e77a825311f55\": container with ID starting with 81cb1590c843dc22421c710b134b517b7ca49efb367815ff446e77a825311f55 not found: ID does not exist" Oct 30 02:46:18.982202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.982162 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg"] Oct 30 02:46:18.992116 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:18.992067 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-2c4b44-openshift-default-79c69c5d7c-sm2kg"] Oct 30 02:46:19.750444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:19.750401 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:20.394178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:20.394143 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:20.545992 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:20.545945 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b13ac864-0406-4db5-b911-2851f86df2b6" path="/var/lib/kubelet/pods/b13ac864-0406-4db5-b911-2851f86df2b6/volumes" Oct 30 02:46:20.582723 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:20.582685 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:20.782500 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:20.782463 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:21.492498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:21.492408 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:22.190956 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:22.190913 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:23.763156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:23.763114 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:25.065934 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:25.065893 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:25.257395 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:46:25.257350 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:46:25.676756 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:25.676720 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:25.834913 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:25.834874 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:46:27.032395 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:27.032355 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc"] Oct 30 02:46:27.041538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:27.041509 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869e9f05-mxpwc"] Oct 30 02:46:27.263864 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:27.263815 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:27.502532 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:27.502483 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:27.621551 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:27.621520 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:27.726254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:27.726219 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:28.185259 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:28.185223 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:28.469274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:28.469174 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:28.551190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:28.551137 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00355226-9a13-4568-8152-76c07ac13698" path="/var/lib/kubelet/pods/00355226-9a13-4568-8152-76c07ac13698/volumes" Oct 30 02:46:28.551607 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:28.551583 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg"] Oct 30 02:46:28.552022 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:28.551990 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" podUID="57be1a46-2149-46f4-a048-b099d67d3a6b" containerName="istio-proxy" containerID="cri-o://32a424da9f0dbaaeef09ef6cd48a253ec4bc55e899d8fdebfeccac9e52eba361" gracePeriod=30 Oct 30 02:46:28.580622 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:28.580581 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:28.706564 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:28.706530 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:29.002661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:29.002576 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:29.124778 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:29.124725 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:29.407687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:29.407560 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:29.731207 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:29.731162 5998 patch_prober.go:29] interesting pod/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:46:29.731385 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:29.731240 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" podUID="57be1a46-2149-46f4-a048-b099d67d3a6b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:46:29.873270 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:29.873217 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:30.422545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:30.422510 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:31.071710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:31.071675 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:32.670926 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:32.670887 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:33.812438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.812409 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:46:33.899858 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.899711 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-podinfo\") pod \"57be1a46-2149-46f4-a048-b099d67d3a6b\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " Oct 30 02:46:33.900082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.899879 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-envoy\") pod \"57be1a46-2149-46f4-a048-b099d67d3a6b\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " Oct 30 02:46:33.900082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.899940 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-data\") pod \"57be1a46-2149-46f4-a048-b099d67d3a6b\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " Oct 30 02:46:33.900082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.899985 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57be1a46-2149-46f4-a048-b099d67d3a6b-kube-api-access-vzmcf\" (UniqueName: \"kubernetes.io/projected/57be1a46-2149-46f4-a048-b099d67d3a6b-kube-api-access-vzmcf\") pod \"57be1a46-2149-46f4-a048-b099d67d3a6b\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " Oct 30 02:46:33.900082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.900020 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-workload-certs\") pod \"57be1a46-2149-46f4-a048-b099d67d3a6b\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " Oct 30 02:46:33.900082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.900063 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-token\" (UniqueName: \"kubernetes.io/projected/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-token\") pod \"57be1a46-2149-46f4-a048-b099d67d3a6b\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " Oct 30 02:46:33.900422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.900169 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-workload-socket\") pod \"57be1a46-2149-46f4-a048-b099d67d3a6b\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " Oct 30 02:46:33.900422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.900209 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57be1a46-2149-46f4-a048-b099d67d3a6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57be1a46-2149-46f4-a048-b099d67d3a6b-istiod-ca-cert\") pod \"57be1a46-2149-46f4-a048-b099d67d3a6b\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " Oct 30 02:46:33.900422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.900253 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-credential-socket\") pod \"57be1a46-2149-46f4-a048-b099d67d3a6b\" (UID: \"57be1a46-2149-46f4-a048-b099d67d3a6b\") " Oct 30 02:46:33.900797 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.900765 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-data" pod "57be1a46-2149-46f4-a048-b099d67d3a6b" (UID: "57be1a46-2149-46f4-a048-b099d67d3a6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:33.900941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.900815 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-workload-socket" pod "57be1a46-2149-46f4-a048-b099d67d3a6b" (UID: "57be1a46-2149-46f4-a048-b099d67d3a6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:33.901225 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.901119 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57be1a46-2149-46f4-a048-b099d67d3a6b-istiod-ca-cert" pod "57be1a46-2149-46f4-a048-b099d67d3a6b" (UID: "57be1a46-2149-46f4-a048-b099d67d3a6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:33.901364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.901198 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-workload-certs" pod "57be1a46-2149-46f4-a048-b099d67d3a6b" (UID: "57be1a46-2149-46f4-a048-b099d67d3a6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:33.901364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.901292 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-credential-socket" pod "57be1a46-2149-46f4-a048-b099d67d3a6b" (UID: "57be1a46-2149-46f4-a048-b099d67d3a6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:33.902742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.902716 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-podinfo" pod "57be1a46-2149-46f4-a048-b099d67d3a6b" (UID: "57be1a46-2149-46f4-a048-b099d67d3a6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:46:33.903015 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.902748 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-envoy" pod "57be1a46-2149-46f4-a048-b099d67d3a6b" (UID: "57be1a46-2149-46f4-a048-b099d67d3a6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:33.903226 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.903193 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-token" pod "57be1a46-2149-46f4-a048-b099d67d3a6b" (UID: "57be1a46-2149-46f4-a048-b099d67d3a6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:33.903312 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:33.903234 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57be1a46-2149-46f4-a048-b099d67d3a6b-kube-api-access-vzmcf" pod "57be1a46-2149-46f4-a048-b099d67d3a6b" (UID: "57be1a46-2149-46f4-a048-b099d67d3a6b"). InnerVolumeSpecName "kube-api-access-vzmcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:34.002477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.002433 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:34.002477 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.002477 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:34.002668 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.002492 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:34.002668 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.002504 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57be1a46-2149-46f4-a048-b099d67d3a6b-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:34.002668 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.002517 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:34.002668 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.002529 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:34.002668 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.002545 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:34.002668 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.002557 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57be1a46-2149-46f4-a048-b099d67d3a6b-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:34.002668 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.002570 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzmcf\" (UniqueName: \"kubernetes.io/projected/57be1a46-2149-46f4-a048-b099d67d3a6b-kube-api-access-vzmcf\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:34.062952 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.062910 5998 generic.go:356] "Generic (PLEG): container finished" podID="57be1a46-2149-46f4-a048-b099d67d3a6b" containerID="32a424da9f0dbaaeef09ef6cd48a253ec4bc55e899d8fdebfeccac9e52eba361" exitCode=0 Oct 30 02:46:34.063156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.062968 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" Oct 30 02:46:34.063156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.062989 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" event={"ID":"57be1a46-2149-46f4-a048-b099d67d3a6b","Type":"ContainerDied","Data":"32a424da9f0dbaaeef09ef6cd48a253ec4bc55e899d8fdebfeccac9e52eba361"} Oct 30 02:46:34.063156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.063048 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg" event={"ID":"57be1a46-2149-46f4-a048-b099d67d3a6b","Type":"ContainerDied","Data":"67a794c4f3766f4cca1a020412dcd45c1b5e1c6a3481f62c64c8a4f3c2098cad"} Oct 30 02:46:34.063156 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.063070 5998 scope.go:122] "RemoveContainer" containerID="32a424da9f0dbaaeef09ef6cd48a253ec4bc55e899d8fdebfeccac9e52eba361" Oct 30 02:46:34.075124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.075106 5998 scope.go:122] "RemoveContainer" containerID="122a350bf9d6bd79a9e50c83b560c5ddf2991e2b97e66d4f77477449c28424b6" Oct 30 02:46:34.085848 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.085824 5998 scope.go:122] "RemoveContainer" containerID="32a424da9f0dbaaeef09ef6cd48a253ec4bc55e899d8fdebfeccac9e52eba361" Oct 30 02:46:34.086182 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:46:34.086162 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a424da9f0dbaaeef09ef6cd48a253ec4bc55e899d8fdebfeccac9e52eba361\": container with ID starting with 32a424da9f0dbaaeef09ef6cd48a253ec4bc55e899d8fdebfeccac9e52eba361 not found: ID does not exist" containerID="32a424da9f0dbaaeef09ef6cd48a253ec4bc55e899d8fdebfeccac9e52eba361" Oct 30 02:46:34.086242 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.086194 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a424da9f0dbaaeef09ef6cd48a253ec4bc55e899d8fdebfeccac9e52eba361"} err="failed to get container status \"32a424da9f0dbaaeef09ef6cd48a253ec4bc55e899d8fdebfeccac9e52eba361\": rpc error: code = NotFound desc = could not find container \"32a424da9f0dbaaeef09ef6cd48a253ec4bc55e899d8fdebfeccac9e52eba361\": container with ID starting with 32a424da9f0dbaaeef09ef6cd48a253ec4bc55e899d8fdebfeccac9e52eba361 not found: ID does not exist" Oct 30 02:46:34.086242 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.086212 5998 scope.go:122] "RemoveContainer" containerID="122a350bf9d6bd79a9e50c83b560c5ddf2991e2b97e66d4f77477449c28424b6" Oct 30 02:46:34.086475 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:46:34.086455 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"122a350bf9d6bd79a9e50c83b560c5ddf2991e2b97e66d4f77477449c28424b6\": container with ID starting with 122a350bf9d6bd79a9e50c83b560c5ddf2991e2b97e66d4f77477449c28424b6 not found: ID does not exist" containerID="122a350bf9d6bd79a9e50c83b560c5ddf2991e2b97e66d4f77477449c28424b6" Oct 30 02:46:34.086516 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.086481 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"122a350bf9d6bd79a9e50c83b560c5ddf2991e2b97e66d4f77477449c28424b6"} err="failed to get container status \"122a350bf9d6bd79a9e50c83b560c5ddf2991e2b97e66d4f77477449c28424b6\": rpc error: code = NotFound desc = could not find container \"122a350bf9d6bd79a9e50c83b560c5ddf2991e2b97e66d4f77477449c28424b6\": container with ID starting with 122a350bf9d6bd79a9e50c83b560c5ddf2991e2b97e66d4f77477449c28424b6 not found: ID does not exist" Oct 30 02:46:34.112926 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.112900 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg"] Oct 30 02:46:34.118907 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.118882 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a0fc73-openshift-default-c964896bf-cx4rg"] Oct 30 02:46:34.277547 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.277510 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:34.540508 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.540421 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57be1a46-2149-46f4-a048-b099d67d3a6b" path="/var/lib/kubelet/pods/57be1a46-2149-46f4-a048-b099d67d3a6b/volumes" Oct 30 02:46:34.989298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:34.989263 5998 scope.go:122] "RemoveContainer" containerID="625e032a79654547409ed683c8678a1c9db2080f89fc507c57a73dd5e3a1fd84" Oct 30 02:46:35.863480 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:35.863444 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:36.457803 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:36.457763 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:36.563250 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:36.563211 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:36.831382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:36.831265 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:36.939606 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:36.939566 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:37.521807 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:37.521754 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:37.655011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:37.654979 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:37.796137 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:37.796025 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:37.923159 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:37.918498 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:37.962454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:37.953447 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str"] Oct 30 02:46:37.962454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:37.953713 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" podUID="29ec48a4-d344-4e9c-bba3-b1b68bc21a84" containerName="istio-proxy" containerID="cri-o://e9d2da0202a128ad214bf1e0c21da65eb30562d4b4d441932dd0f6bc7621096e" gracePeriod=30 Oct 30 02:46:38.035750 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:38.035702 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:38.781320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:38.781281 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:38.944210 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:38.944163 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:39.373289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:39.373254 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:40.627554 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:40.627515 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:46:40.965433 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:40.965397 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:41.626498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:41.626455 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:42.212383 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:42.212348 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:42.330502 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:42.330464 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:42.512786 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:42.512694 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:43.073450 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.073417 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:43.131399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.131361 5998 generic.go:356] "Generic (PLEG): container finished" podID="29ec48a4-d344-4e9c-bba3-b1b68bc21a84" containerID="e9d2da0202a128ad214bf1e0c21da65eb30562d4b4d441932dd0f6bc7621096e" exitCode=0 Oct 30 02:46:43.131718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.131411 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" event={"ID":"29ec48a4-d344-4e9c-bba3-b1b68bc21a84","Type":"ContainerDied","Data":"e9d2da0202a128ad214bf1e0c21da65eb30562d4b4d441932dd0f6bc7621096e"} Oct 30 02:46:43.131805 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.131765 5998 scope.go:122] "RemoveContainer" containerID="46d21d37f9becc1c0ef4e0a192c45cbe5dd8bfd2a9fb4829ac9d77ee63132ce5" Oct 30 02:46:43.254870 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.254797 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:46:43.411678 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.411636 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-workload-certs\") pod \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " Oct 30 02:46:43.411893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.411724 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istiod-ca-cert\") pod \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " Oct 30 02:46:43.411893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.411756 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-token\" (UniqueName: \"kubernetes.io/projected/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-token\") pod \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " Oct 30 02:46:43.411893 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.411850 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-podinfo\") pod \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " Oct 30 02:46:43.412129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.411953 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-workload-certs" pod "29ec48a4-d344-4e9c-bba3-b1b68bc21a84" (UID: "29ec48a4-d344-4e9c-bba3-b1b68bc21a84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:43.412129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.411967 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-envoy\") pod \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " Oct 30 02:46:43.412129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.412059 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-kube-api-access-kph7q\" (UniqueName: \"kubernetes.io/projected/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-kube-api-access-kph7q\") pod \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " Oct 30 02:46:43.412129 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.412118 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-credential-socket\") pod \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " Oct 30 02:46:43.412393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.412166 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-data\") pod \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " Oct 30 02:46:43.412393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.412202 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-workload-socket\") pod \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\" (UID: \"29ec48a4-d344-4e9c-bba3-b1b68bc21a84\") " Oct 30 02:46:43.412393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.412235 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istiod-ca-cert" pod "29ec48a4-d344-4e9c-bba3-b1b68bc21a84" (UID: "29ec48a4-d344-4e9c-bba3-b1b68bc21a84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:43.412563 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.412536 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-credential-socket" pod "29ec48a4-d344-4e9c-bba3-b1b68bc21a84" (UID: "29ec48a4-d344-4e9c-bba3-b1b68bc21a84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:43.412664 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.412634 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-data" pod "29ec48a4-d344-4e9c-bba3-b1b68bc21a84" (UID: "29ec48a4-d344-4e9c-bba3-b1b68bc21a84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:43.412737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.412702 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:43.412737 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.412726 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:43.412869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.412740 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:43.412869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.412748 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-workload-socket" pod "29ec48a4-d344-4e9c-bba3-b1b68bc21a84" (UID: "29ec48a4-d344-4e9c-bba3-b1b68bc21a84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:43.414491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.414455 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-token" pod "29ec48a4-d344-4e9c-bba3-b1b68bc21a84" (UID: "29ec48a4-d344-4e9c-bba3-b1b68bc21a84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:43.414610 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.414555 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-podinfo" pod "29ec48a4-d344-4e9c-bba3-b1b68bc21a84" (UID: "29ec48a4-d344-4e9c-bba3-b1b68bc21a84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:46:43.414800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.414780 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-envoy" pod "29ec48a4-d344-4e9c-bba3-b1b68bc21a84" (UID: "29ec48a4-d344-4e9c-bba3-b1b68bc21a84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:43.414870 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.414821 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-kube-api-access-kph7q" pod "29ec48a4-d344-4e9c-bba3-b1b68bc21a84" (UID: "29ec48a4-d344-4e9c-bba3-b1b68bc21a84"). InnerVolumeSpecName "kube-api-access-kph7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:43.514029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.513934 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:43.514029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.513975 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:43.514029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.513986 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:43.514029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.513994 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kph7q\" (UniqueName: \"kubernetes.io/projected/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-kube-api-access-kph7q\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:43.514029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.514006 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:43.514029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:43.514014 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ec48a4-d344-4e9c-bba3-b1b68bc21a84-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:44.141006 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:44.140978 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" Oct 30 02:46:44.141006 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:44.140986 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str" event={"ID":"29ec48a4-d344-4e9c-bba3-b1b68bc21a84","Type":"ContainerDied","Data":"a41dc79e82a01b9ebd8ed5b1c63a3fccec3b30603612279083096faa1e0d20e5"} Oct 30 02:46:44.141289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:44.141045 5998 scope.go:122] "RemoveContainer" containerID="e9d2da0202a128ad214bf1e0c21da65eb30562d4b4d441932dd0f6bc7621096e" Oct 30 02:46:44.196126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:44.196070 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str"] Oct 30 02:46:44.205866 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:44.205833 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-584ec5-openshift-default-df7b9d5ff-h8str"] Oct 30 02:46:44.538311 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:44.538273 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29ec48a4-d344-4e9c-bba3-b1b68bc21a84" path="/var/lib/kubelet/pods/29ec48a4-d344-4e9c-bba3-b1b68bc21a84/volumes" Oct 30 02:46:44.673894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:44.673856 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:45.126068 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:45.126031 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:45.231679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:45.231637 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:45.502219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:45.502182 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:45.647015 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:45.646979 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:46.175257 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:46.175223 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:46.376410 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:46.376373 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:46.559976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:46.559879 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh"] Oct 30 02:46:46.560183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:46.560118 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" podUID="eef42836-3659-4fdc-b2ff-b8003e3e47a3" containerName="istio-proxy" containerID="cri-o://4d81353fec7c40562c57163a6ac0d8fba1930d85d918a640fd769c1711949972" gracePeriod=30 Oct 30 02:46:46.565272 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:46.565244 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:46.882249 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:46.882164 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:47.375253 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:47.375221 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:47.784941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:47.784899 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:47.895552 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:47.895516 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:48.010482 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:48.010440 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:48.040970 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:48.040881 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh"] Oct 30 02:46:48.050646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:48.050609 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b7a1b68-w7glh"] Oct 30 02:46:48.332384 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:48.332288 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:48.538732 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:48.538697 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b5bea28-5420-4913-a4bb-eb8bbf18dbb7" path="/var/lib/kubelet/pods/8b5bea28-5420-4913-a4bb-eb8bbf18dbb7/volumes" Oct 30 02:46:48.593298 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:48.593218 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:49.972354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:49.972303 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:50.585053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:50.585015 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:51.826833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:51.826804 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:46:52.010687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.010648 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-podinfo\") pod \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " Oct 30 02:46:52.010949 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.010709 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-data\") pod \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " Oct 30 02:46:52.010949 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.010757 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-token\" (UniqueName: \"kubernetes.io/projected/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-token\") pod \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " Oct 30 02:46:52.010949 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.010840 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-workload-socket\") pod \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " Oct 30 02:46:52.010949 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.010875 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-credential-socket\") pod \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " Oct 30 02:46:52.011219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.011192 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-data" pod "eef42836-3659-4fdc-b2ff-b8003e3e47a3" (UID: "eef42836-3659-4fdc-b2ff-b8003e3e47a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:52.011292 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.011205 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-workload-socket" pod "eef42836-3659-4fdc-b2ff-b8003e3e47a3" (UID: "eef42836-3659-4fdc-b2ff-b8003e3e47a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:52.011338 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.011301 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istiod-ca-cert\") pod \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " Oct 30 02:46:52.011376 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.011361 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef42836-3659-4fdc-b2ff-b8003e3e47a3-kube-api-access-bfnm7\" (UniqueName: \"kubernetes.io/projected/eef42836-3659-4fdc-b2ff-b8003e3e47a3-kube-api-access-bfnm7\") pod \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " Oct 30 02:46:52.011424 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.011299 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-credential-socket" pod "eef42836-3659-4fdc-b2ff-b8003e3e47a3" (UID: "eef42836-3659-4fdc-b2ff-b8003e3e47a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:52.011424 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.011418 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-envoy\") pod \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " Oct 30 02:46:52.011539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.011503 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-workload-certs\") pod \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\" (UID: \"eef42836-3659-4fdc-b2ff-b8003e3e47a3\") " Oct 30 02:46:52.011597 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.011572 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istiod-ca-cert" pod "eef42836-3659-4fdc-b2ff-b8003e3e47a3" (UID: "eef42836-3659-4fdc-b2ff-b8003e3e47a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:52.011925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.011893 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-workload-certs" pod "eef42836-3659-4fdc-b2ff-b8003e3e47a3" (UID: "eef42836-3659-4fdc-b2ff-b8003e3e47a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:52.012315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.012294 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:52.012414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.012320 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:52.012414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.012334 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:52.012414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.012347 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:52.012414 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.012360 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:52.013973 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.013948 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-envoy" pod "eef42836-3659-4fdc-b2ff-b8003e3e47a3" (UID: "eef42836-3659-4fdc-b2ff-b8003e3e47a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:46:52.013973 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.013956 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-token" pod "eef42836-3659-4fdc-b2ff-b8003e3e47a3" (UID: "eef42836-3659-4fdc-b2ff-b8003e3e47a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:52.014427 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.014409 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef42836-3659-4fdc-b2ff-b8003e3e47a3-kube-api-access-bfnm7" pod "eef42836-3659-4fdc-b2ff-b8003e3e47a3" (UID: "eef42836-3659-4fdc-b2ff-b8003e3e47a3"). InnerVolumeSpecName "kube-api-access-bfnm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:52.014581 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.014558 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-podinfo" pod "eef42836-3659-4fdc-b2ff-b8003e3e47a3" (UID: "eef42836-3659-4fdc-b2ff-b8003e3e47a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:46:52.078833 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.078798 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:52.113416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.113379 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:52.113416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.113413 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:52.113671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.113427 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfnm7\" (UniqueName: \"kubernetes.io/projected/eef42836-3659-4fdc-b2ff-b8003e3e47a3-kube-api-access-bfnm7\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:52.113671 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.113444 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef42836-3659-4fdc-b2ff-b8003e3e47a3-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:52.204014 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.203975 5998 generic.go:356] "Generic (PLEG): container finished" podID="eef42836-3659-4fdc-b2ff-b8003e3e47a3" containerID="4d81353fec7c40562c57163a6ac0d8fba1930d85d918a640fd769c1711949972" exitCode=0 Oct 30 02:46:52.204014 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.204024 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" Oct 30 02:46:52.204328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.204060 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" event={"ID":"eef42836-3659-4fdc-b2ff-b8003e3e47a3","Type":"ContainerDied","Data":"4d81353fec7c40562c57163a6ac0d8fba1930d85d918a640fd769c1711949972"} Oct 30 02:46:52.204328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.204133 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh" event={"ID":"eef42836-3659-4fdc-b2ff-b8003e3e47a3","Type":"ContainerDied","Data":"c48cc7c3f1f631e2b2137c51a3100345b7dc37aa2dc71dfa5700ffac22d0b804"} Oct 30 02:46:52.204328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.204167 5998 scope.go:122] "RemoveContainer" containerID="4d81353fec7c40562c57163a6ac0d8fba1930d85d918a640fd769c1711949972" Oct 30 02:46:52.219192 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.219170 5998 scope.go:122] "RemoveContainer" containerID="36eb90a21df0e911eec233e8ffa2718e87689656261903f0b350f12f26b420b4" Oct 30 02:46:52.228488 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.228467 5998 scope.go:122] "RemoveContainer" containerID="4d81353fec7c40562c57163a6ac0d8fba1930d85d918a640fd769c1711949972" Oct 30 02:46:52.228800 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:46:52.228780 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d81353fec7c40562c57163a6ac0d8fba1930d85d918a640fd769c1711949972\": container with ID starting with 4d81353fec7c40562c57163a6ac0d8fba1930d85d918a640fd769c1711949972 not found: ID does not exist" containerID="4d81353fec7c40562c57163a6ac0d8fba1930d85d918a640fd769c1711949972" Oct 30 02:46:52.228885 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.228808 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d81353fec7c40562c57163a6ac0d8fba1930d85d918a640fd769c1711949972"} err="failed to get container status \"4d81353fec7c40562c57163a6ac0d8fba1930d85d918a640fd769c1711949972\": rpc error: code = NotFound desc = could not find container \"4d81353fec7c40562c57163a6ac0d8fba1930d85d918a640fd769c1711949972\": container with ID starting with 4d81353fec7c40562c57163a6ac0d8fba1930d85d918a640fd769c1711949972 not found: ID does not exist" Oct 30 02:46:52.228885 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.228825 5998 scope.go:122] "RemoveContainer" containerID="36eb90a21df0e911eec233e8ffa2718e87689656261903f0b350f12f26b420b4" Oct 30 02:46:52.229135 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:46:52.229113 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36eb90a21df0e911eec233e8ffa2718e87689656261903f0b350f12f26b420b4\": container with ID starting with 36eb90a21df0e911eec233e8ffa2718e87689656261903f0b350f12f26b420b4 not found: ID does not exist" containerID="36eb90a21df0e911eec233e8ffa2718e87689656261903f0b350f12f26b420b4" Oct 30 02:46:52.229208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.229143 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36eb90a21df0e911eec233e8ffa2718e87689656261903f0b350f12f26b420b4"} err="failed to get container status \"36eb90a21df0e911eec233e8ffa2718e87689656261903f0b350f12f26b420b4\": rpc error: code = NotFound desc = could not find container \"36eb90a21df0e911eec233e8ffa2718e87689656261903f0b350f12f26b420b4\": container with ID starting with 36eb90a21df0e911eec233e8ffa2718e87689656261903f0b350f12f26b420b4 not found: ID does not exist" Oct 30 02:46:52.252656 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.252621 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh"] Oct 30 02:46:52.259503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.259471 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0fdc4e-openshift-default-677dc68fd9-g7rlh"] Oct 30 02:46:52.538345 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:52.538309 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eef42836-3659-4fdc-b2ff-b8003e3e47a3" path="/var/lib/kubelet/pods/eef42836-3659-4fdc-b2ff-b8003e3e47a3/volumes" Oct 30 02:46:53.780866 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:53.780820 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:53.914146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:53.914112 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:54.030440 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:54.030395 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:54.296534 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:54.296498 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:54.419280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:54.419240 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:54.615082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:54.614999 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:55.034161 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.034114 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:55.162242 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.162206 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:55.223044 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:46:55.223004 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:46:55.479388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.479358 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:55.589203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.589162 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:55.915722 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.915687 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn"] Oct 30 02:46:55.916128 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916111 5998 cpu_manager.go:414] "containerMap: removing container" podUID="acb7072a-7343-4122-9472-93c1a658cb02" containerName="istio-proxy" Oct 30 02:46:55.916185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916131 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb7072a-7343-4122-9472-93c1a658cb02" containerName="istio-proxy" Oct 30 02:46:55.916185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916147 5998 cpu_manager.go:414] "containerMap: removing container" podUID="57be1a46-2149-46f4-a048-b099d67d3a6b" containerName="istio-proxy" Oct 30 02:46:55.916185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916153 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57be1a46-2149-46f4-a048-b099d67d3a6b" containerName="istio-proxy" Oct 30 02:46:55.916185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916161 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7076db80-f9a3-423f-b198-f0a7aba80bd7" containerName="istio-proxy" Oct 30 02:46:55.916185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916167 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7076db80-f9a3-423f-b198-f0a7aba80bd7" containerName="istio-proxy" Oct 30 02:46:55.916185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916179 5998 cpu_manager.go:414] "containerMap: removing container" podUID="b13ac864-0406-4db5-b911-2851f86df2b6" containerName="istio-proxy" Oct 30 02:46:55.916185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916183 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b13ac864-0406-4db5-b911-2851f86df2b6" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916191 5998 cpu_manager.go:414] "containerMap: removing container" podUID="29ec48a4-d344-4e9c-bba3-b1b68bc21a84" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916196 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ec48a4-d344-4e9c-bba3-b1b68bc21a84" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916202 5998 cpu_manager.go:414] "containerMap: removing container" podUID="b13ac864-0406-4db5-b911-2851f86df2b6" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916207 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b13ac864-0406-4db5-b911-2851f86df2b6" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916213 5998 cpu_manager.go:414] "containerMap: removing container" podUID="b9f59989-527d-411d-9fc0-ce758b48e18a" containerName="cleanup" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916217 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f59989-527d-411d-9fc0-ce758b48e18a" containerName="cleanup" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916227 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916232 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916240 5998 cpu_manager.go:414] "containerMap: removing container" podUID="29ec48a4-d344-4e9c-bba3-b1b68bc21a84" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916245 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ec48a4-d344-4e9c-bba3-b1b68bc21a84" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916253 5998 cpu_manager.go:414] "containerMap: removing container" podUID="7076db80-f9a3-423f-b198-f0a7aba80bd7" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916257 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7076db80-f9a3-423f-b198-f0a7aba80bd7" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916265 5998 cpu_manager.go:414] "containerMap: removing container" podUID="af31090b-f48b-4675-9add-665d0f5d42af" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916269 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af31090b-f48b-4675-9add-665d0f5d42af" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916274 5998 cpu_manager.go:414] "containerMap: removing container" podUID="94e4cea7-0d7d-46b1-810b-4567363be4eb" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916280 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e4cea7-0d7d-46b1-810b-4567363be4eb" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916288 5998 cpu_manager.go:414] "containerMap: removing container" podUID="dffe7505-ba98-42c5-aba9-0ef9539b16e4" containerName="revoke-keys" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916293 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dffe7505-ba98-42c5-aba9-0ef9539b16e4" containerName="revoke-keys" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916301 5998 cpu_manager.go:414] "containerMap: removing container" podUID="42fa7bf8-25e0-4bd8-b751-243b12d986b3" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916306 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fa7bf8-25e0-4bd8-b751-243b12d986b3" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916312 5998 cpu_manager.go:414] "containerMap: removing container" podUID="5f859aa2-e664-4e9b-b8d7-474991da223d" containerName="main" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916316 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f859aa2-e664-4e9b-b8d7-474991da223d" containerName="main" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916323 5998 cpu_manager.go:414] "containerMap: removing container" podUID="eef42836-3659-4fdc-b2ff-b8003e3e47a3" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916327 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eef42836-3659-4fdc-b2ff-b8003e3e47a3" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916333 5998 cpu_manager.go:414] "containerMap: removing container" podUID="13bc0eed-818a-4ef8-bc7a-6d9a2aed513f" containerName="main" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916337 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13bc0eed-818a-4ef8-bc7a-6d9a2aed513f" containerName="main" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916344 5998 cpu_manager.go:414] "containerMap: removing container" podUID="662d5231-266e-4ff9-8e5a-4409da2c7d82" containerName="cleanup" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916349 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="662d5231-266e-4ff9-8e5a-4409da2c7d82" containerName="cleanup" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916360 5998 cpu_manager.go:414] "containerMap: removing container" podUID="4a0058da-63f6-48d6-9d96-316956c55dd4" containerName="cleanup" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916365 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0058da-63f6-48d6-9d96-316956c55dd4" containerName="cleanup" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916371 5998 cpu_manager.go:414] "containerMap: removing container" podUID="cce212c5-9e99-4e58-8dde-89b23f8e9e2e" containerName="cleanup" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916376 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce212c5-9e99-4e58-8dde-89b23f8e9e2e" containerName="cleanup" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916403 5998 cpu_manager.go:414] "containerMap: removing container" podUID="eef42836-3659-4fdc-b2ff-b8003e3e47a3" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916410 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eef42836-3659-4fdc-b2ff-b8003e3e47a3" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916423 5998 cpu_manager.go:414] "containerMap: removing container" podUID="af31090b-f48b-4675-9add-665d0f5d42af" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916428 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af31090b-f48b-4675-9add-665d0f5d42af" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916497 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="b13ac864-0406-4db5-b911-2851f86df2b6" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916505 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c560bec5-d0f1-4854-8ca0-24d44f73c750" containerName="cleanup" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916513 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3ed4ad2-da4e-4ed2-bebe-f64b80ea093b" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916521 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ec48a4-d344-4e9c-bba3-b1b68bc21a84" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916527 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7076db80-f9a3-423f-b198-f0a7aba80bd7" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916533 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="af31090b-f48b-4675-9add-665d0f5d42af" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916538 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e4cea7-0d7d-46b1-810b-4567363be4eb" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916544 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="dffe7505-ba98-42c5-aba9-0ef9539b16e4" containerName="revoke-keys" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916549 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="42fa7bf8-25e0-4bd8-b751-243b12d986b3" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916554 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f859aa2-e664-4e9b-b8d7-474991da223d" containerName="main" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916560 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="eef42836-3659-4fdc-b2ff-b8003e3e47a3" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916565 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="13bc0eed-818a-4ef8-bc7a-6d9a2aed513f" containerName="main" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916573 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="eef42836-3659-4fdc-b2ff-b8003e3e47a3" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916579 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="57be1a46-2149-46f4-a048-b099d67d3a6b" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916584 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="af31090b-f48b-4675-9add-665d0f5d42af" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916590 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="acb7072a-7343-4122-9472-93c1a658cb02" containerName="istio-proxy" Oct 30 02:46:55.916731 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.916596 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ec48a4-d344-4e9c-bba3-b1b68bc21a84" containerName="istio-proxy" Oct 30 02:46:55.921347 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.921327 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn" Oct 30 02:46:55.934760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.934735 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gmz8c\"" Oct 30 02:46:55.935443 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.935423 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:46:55.945315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.945290 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:46:55.948526 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.948499 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn"] Oct 30 02:46:55.959968 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.959933 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npgw5\" (UniqueName: \"kubernetes.io/projected/10593def-555a-449b-9da3-597a153c257d-kube-api-access-npgw5\") pod \"maas-api-revoke-sub-16862d1f-4qhtn\" (UID: \"10593def-555a-449b-9da3-597a153c257d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn" Oct 30 02:46:55.960127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.959994 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10593def-555a-449b-9da3-597a153c257d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16862d1f-4qhtn\" (UID: \"10593def-555a-449b-9da3-597a153c257d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn" Oct 30 02:46:55.960127 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:55.960073 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10593def-555a-449b-9da3-597a153c257d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16862d1f-4qhtn\" (UID: \"10593def-555a-449b-9da3-597a153c257d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn" Oct 30 02:46:56.035274 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:56.035233 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:46:56.061081 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:56.061032 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npgw5\" (UniqueName: \"kubernetes.io/projected/10593def-555a-449b-9da3-597a153c257d-kube-api-access-npgw5\") pod \"maas-api-revoke-sub-16862d1f-4qhtn\" (UID: \"10593def-555a-449b-9da3-597a153c257d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn" Oct 30 02:46:56.061328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:56.061139 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10593def-555a-449b-9da3-597a153c257d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16862d1f-4qhtn\" (UID: \"10593def-555a-449b-9da3-597a153c257d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn" Oct 30 02:46:56.061328 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:56.061189 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10593def-555a-449b-9da3-597a153c257d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16862d1f-4qhtn\" (UID: \"10593def-555a-449b-9da3-597a153c257d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn" Oct 30 02:46:56.061925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:56.061895 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10593def-555a-449b-9da3-597a153c257d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16862d1f-4qhtn\" (UID: \"10593def-555a-449b-9da3-597a153c257d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn" Oct 30 02:46:56.063673 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:56.063646 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10593def-555a-449b-9da3-597a153c257d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16862d1f-4qhtn\" (UID: \"10593def-555a-449b-9da3-597a153c257d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn" Oct 30 02:46:56.082399 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:56.082361 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npgw5\" (UniqueName: \"kubernetes.io/projected/10593def-555a-449b-9da3-597a153c257d-kube-api-access-npgw5\") pod \"maas-api-revoke-sub-16862d1f-4qhtn\" (UID: \"10593def-555a-449b-9da3-597a153c257d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn" Oct 30 02:46:56.199695 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:56.199616 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:56.234818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:56.234784 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn" Oct 30 02:46:56.309254 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:56.309157 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:56.545544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:56.545298 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn"] Oct 30 02:46:56.548462 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:46:56.548418 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10593def_555a_449b_9da3_597a153c257d.slice/crio-cf882ef680b47913411c1676f6d2bb2236df5a4172f53662abc49553d1b0378d WatchSource:0}: Error finding container cf882ef680b47913411c1676f6d2bb2236df5a4172f53662abc49553d1b0378d: Status 404 returned error can't find the container with id cf882ef680b47913411c1676f6d2bb2236df5a4172f53662abc49553d1b0378d Oct 30 02:46:56.889774 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:56.889720 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:57.260084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:57.260043 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn" event={"ID":"10593def-555a-449b-9da3-597a153c257d","Type":"ContainerStarted","Data":"a7f9b22aba27325fcbabb1419c6bbace600e869eed200c1003751ccfcea6b3fa"} Oct 30 02:46:57.260633 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:57.260136 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn" event={"ID":"10593def-555a-449b-9da3-597a153c257d","Type":"ContainerStarted","Data":"cf882ef680b47913411c1676f6d2bb2236df5a4172f53662abc49553d1b0378d"} Oct 30 02:46:57.275025 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:57.274988 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:57.308044 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:57.307998 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9"] Oct 30 02:46:57.308479 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:57.308277 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" podUID="820068d2-ba3f-4127-8f01-3b21c61acebd" containerName="istio-proxy" containerID="cri-o://122a307ed172d34a716059cf3da2d2e8f928796402ad64b1a67c55b0e3dc6084" gracePeriod=30 Oct 30 02:46:57.321042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:57.320974 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn" podStartSLOduration=2.320956022 podStartE2EDuration="2.320956022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:46:57.303574157 +0000 UTC m=+4646.882365242" watchObservedRunningTime="2026-10-30 02:46:57.320956022 +0000 UTC m=+4646.899747106" Oct 30 02:46:57.491659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:57.491617 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:58.161163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:58.161125 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:58.267741 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:58.267704 5998 generic.go:356] "Generic (PLEG): container finished" podID="10593def-555a-449b-9da3-597a153c257d" containerID="a7f9b22aba27325fcbabb1419c6bbace600e869eed200c1003751ccfcea6b3fa" exitCode=0 Oct 30 02:46:58.268165 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:58.267788 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn" event={"ID":"10593def-555a-449b-9da3-597a153c257d","Type":"ContainerDied","Data":"a7f9b22aba27325fcbabb1419c6bbace600e869eed200c1003751ccfcea6b3fa"} Oct 30 02:46:58.956151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:58.956113 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:46:59.425712 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:59.425686 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn" Oct 30 02:46:59.498026 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:59.497989 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10593def-555a-449b-9da3-597a153c257d-kube-api-access-npgw5\" (UniqueName: \"kubernetes.io/projected/10593def-555a-449b-9da3-597a153c257d-kube-api-access-npgw5\") pod \"10593def-555a-449b-9da3-597a153c257d\" (UID: \"10593def-555a-449b-9da3-597a153c257d\") " Oct 30 02:46:59.498247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:59.498132 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10593def-555a-449b-9da3-597a153c257d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10593def-555a-449b-9da3-597a153c257d-maas-api-xks-ca\") pod \"10593def-555a-449b-9da3-597a153c257d\" (UID: \"10593def-555a-449b-9da3-597a153c257d\") " Oct 30 02:46:59.498247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:59.498231 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10593def-555a-449b-9da3-597a153c257d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10593def-555a-449b-9da3-597a153c257d-maas-api-openshift-ca\") pod \"10593def-555a-449b-9da3-597a153c257d\" (UID: \"10593def-555a-449b-9da3-597a153c257d\") " Oct 30 02:46:59.498658 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:59.498630 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10593def-555a-449b-9da3-597a153c257d-maas-api-openshift-ca" pod "10593def-555a-449b-9da3-597a153c257d" (UID: "10593def-555a-449b-9da3-597a153c257d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:46:59.500208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:59.500182 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10593def-555a-449b-9da3-597a153c257d-kube-api-access-npgw5" pod "10593def-555a-449b-9da3-597a153c257d" (UID: "10593def-555a-449b-9da3-597a153c257d"). InnerVolumeSpecName "kube-api-access-npgw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:46:59.500208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:59.500190 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10593def-555a-449b-9da3-597a153c257d-maas-api-xks-ca" pod "10593def-555a-449b-9da3-597a153c257d" (UID: "10593def-555a-449b-9da3-597a153c257d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:46:59.600047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:59.599969 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10593def-555a-449b-9da3-597a153c257d-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:59.600047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:59.600001 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10593def-555a-449b-9da3-597a153c257d-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:46:59.600047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:46:59.600012 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npgw5\" (UniqueName: \"kubernetes.io/projected/10593def-555a-449b-9da3-597a153c257d-kube-api-access-npgw5\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:00.286427 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:00.286366 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn" event={"ID":"10593def-555a-449b-9da3-597a153c257d","Type":"ContainerDied","Data":"cf882ef680b47913411c1676f6d2bb2236df5a4172f53662abc49553d1b0378d"} Oct 30 02:47:00.286702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:00.286451 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf882ef680b47913411c1676f6d2bb2236df5a4172f53662abc49553d1b0378d" Oct 30 02:47:00.286702 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:00.286478 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn" Oct 30 02:47:00.341256 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:00.341153 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:00.563959 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:00.563873 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:01.080286 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:01.080245 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:01.328120 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:01.328060 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:01.961941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:01.961901 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:02.070157 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.070122 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:02.582220 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.582156 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:47:02.635516 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.635478 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-workload-certs\") pod \"820068d2-ba3f-4127-8f01-3b21c61acebd\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " Oct 30 02:47:02.635516 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.635521 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-data\") pod \"820068d2-ba3f-4127-8f01-3b21c61acebd\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " Oct 30 02:47:02.635765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.635550 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-token\" (UniqueName: \"kubernetes.io/projected/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-token\") pod \"820068d2-ba3f-4127-8f01-3b21c61acebd\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " Oct 30 02:47:02.635765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.635581 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/820068d2-ba3f-4127-8f01-3b21c61acebd-kube-api-access-nnq8g\" (UniqueName: \"kubernetes.io/projected/820068d2-ba3f-4127-8f01-3b21c61acebd-kube-api-access-nnq8g\") pod \"820068d2-ba3f-4127-8f01-3b21c61acebd\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " Oct 30 02:47:02.635765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.635617 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-podinfo\") pod \"820068d2-ba3f-4127-8f01-3b21c61acebd\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " Oct 30 02:47:02.635765 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.635663 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-workload-socket\") pod \"820068d2-ba3f-4127-8f01-3b21c61acebd\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " Oct 30 02:47:02.635990 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.635779 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-envoy\") pod \"820068d2-ba3f-4127-8f01-3b21c61acebd\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " Oct 30 02:47:02.635990 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.635815 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-credential-socket\") pod \"820068d2-ba3f-4127-8f01-3b21c61acebd\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " Oct 30 02:47:02.635990 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.635857 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/820068d2-ba3f-4127-8f01-3b21c61acebd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/820068d2-ba3f-4127-8f01-3b21c61acebd-istiod-ca-cert\") pod \"820068d2-ba3f-4127-8f01-3b21c61acebd\" (UID: \"820068d2-ba3f-4127-8f01-3b21c61acebd\") " Oct 30 02:47:02.635990 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.635877 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-workload-certs" pod "820068d2-ba3f-4127-8f01-3b21c61acebd" (UID: "820068d2-ba3f-4127-8f01-3b21c61acebd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:02.636679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.636324 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:02.636679 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.636452 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-data" pod "820068d2-ba3f-4127-8f01-3b21c61acebd" (UID: "820068d2-ba3f-4127-8f01-3b21c61acebd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:02.636900 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.636786 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-workload-socket" pod "820068d2-ba3f-4127-8f01-3b21c61acebd" (UID: "820068d2-ba3f-4127-8f01-3b21c61acebd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:02.637117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.637055 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-credential-socket" pod "820068d2-ba3f-4127-8f01-3b21c61acebd" (UID: "820068d2-ba3f-4127-8f01-3b21c61acebd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:02.637239 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.637173 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/820068d2-ba3f-4127-8f01-3b21c61acebd-istiod-ca-cert" pod "820068d2-ba3f-4127-8f01-3b21c61acebd" (UID: "820068d2-ba3f-4127-8f01-3b21c61acebd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:02.639372 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.639345 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820068d2-ba3f-4127-8f01-3b21c61acebd-kube-api-access-nnq8g" pod "820068d2-ba3f-4127-8f01-3b21c61acebd" (UID: "820068d2-ba3f-4127-8f01-3b21c61acebd"). InnerVolumeSpecName "kube-api-access-nnq8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:02.639497 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.639407 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-token" pod "820068d2-ba3f-4127-8f01-3b21c61acebd" (UID: "820068d2-ba3f-4127-8f01-3b21c61acebd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:02.639662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.639628 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-podinfo" pod "820068d2-ba3f-4127-8f01-3b21c61acebd" (UID: "820068d2-ba3f-4127-8f01-3b21c61acebd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:47:02.639770 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.639710 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-envoy" pod "820068d2-ba3f-4127-8f01-3b21c61acebd" (UID: "820068d2-ba3f-4127-8f01-3b21c61acebd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:02.702082 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.702025 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:02.738841 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.738801 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:02.738841 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.738839 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:02.738841 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.738853 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:02.738841 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.738861 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:02.738841 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.738870 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/820068d2-ba3f-4127-8f01-3b21c61acebd-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:02.739288 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.738880 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:02.739288 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.738888 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/820068d2-ba3f-4127-8f01-3b21c61acebd-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:02.739288 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.738900 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnq8g\" (UniqueName: \"kubernetes.io/projected/820068d2-ba3f-4127-8f01-3b21c61acebd-kube-api-access-nnq8g\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:02.812467 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:02.812420 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:03.260644 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:03.260604 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:03.308002 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:03.307957 5998 generic.go:356] "Generic (PLEG): container finished" podID="820068d2-ba3f-4127-8f01-3b21c61acebd" containerID="122a307ed172d34a716059cf3da2d2e8f928796402ad64b1a67c55b0e3dc6084" exitCode=0 Oct 30 02:47:03.308228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:03.308021 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" Oct 30 02:47:03.308228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:03.308029 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" event={"ID":"820068d2-ba3f-4127-8f01-3b21c61acebd","Type":"ContainerDied","Data":"122a307ed172d34a716059cf3da2d2e8f928796402ad64b1a67c55b0e3dc6084"} Oct 30 02:47:03.308228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:03.308124 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9" event={"ID":"820068d2-ba3f-4127-8f01-3b21c61acebd","Type":"ContainerDied","Data":"9be56fcfb5d4dd4d2fd9daea6b91c06d140a5d6917874f9d435125b1ed9c1ffb"} Oct 30 02:47:03.308228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:03.308160 5998 scope.go:122] "RemoveContainer" containerID="122a307ed172d34a716059cf3da2d2e8f928796402ad64b1a67c55b0e3dc6084" Oct 30 02:47:03.357150 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:03.354685 5998 scope.go:122] "RemoveContainer" containerID="6d618c4cef412e22402f92cbb9bba4fc0122b35c48d9dd47351e462a728d8c76" Oct 30 02:47:03.367456 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:03.367433 5998 scope.go:122] "RemoveContainer" containerID="122a307ed172d34a716059cf3da2d2e8f928796402ad64b1a67c55b0e3dc6084" Oct 30 02:47:03.367706 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:47:03.367683 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"122a307ed172d34a716059cf3da2d2e8f928796402ad64b1a67c55b0e3dc6084\": container with ID starting with 122a307ed172d34a716059cf3da2d2e8f928796402ad64b1a67c55b0e3dc6084 not found: ID does not exist" containerID="122a307ed172d34a716059cf3da2d2e8f928796402ad64b1a67c55b0e3dc6084" Oct 30 02:47:03.367794 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:03.367717 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"122a307ed172d34a716059cf3da2d2e8f928796402ad64b1a67c55b0e3dc6084"} err="failed to get container status \"122a307ed172d34a716059cf3da2d2e8f928796402ad64b1a67c55b0e3dc6084\": rpc error: code = NotFound desc = could not find container \"122a307ed172d34a716059cf3da2d2e8f928796402ad64b1a67c55b0e3dc6084\": container with ID starting with 122a307ed172d34a716059cf3da2d2e8f928796402ad64b1a67c55b0e3dc6084 not found: ID does not exist" Oct 30 02:47:03.367794 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:03.367740 5998 scope.go:122] "RemoveContainer" containerID="6d618c4cef412e22402f92cbb9bba4fc0122b35c48d9dd47351e462a728d8c76" Oct 30 02:47:03.367995 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:47:03.367975 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d618c4cef412e22402f92cbb9bba4fc0122b35c48d9dd47351e462a728d8c76\": container with ID starting with 6d618c4cef412e22402f92cbb9bba4fc0122b35c48d9dd47351e462a728d8c76 not found: ID does not exist" containerID="6d618c4cef412e22402f92cbb9bba4fc0122b35c48d9dd47351e462a728d8c76" Oct 30 02:47:03.368075 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:03.368000 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d618c4cef412e22402f92cbb9bba4fc0122b35c48d9dd47351e462a728d8c76"} err="failed to get container status \"6d618c4cef412e22402f92cbb9bba4fc0122b35c48d9dd47351e462a728d8c76\": rpc error: code = NotFound desc = could not find container \"6d618c4cef412e22402f92cbb9bba4fc0122b35c48d9dd47351e462a728d8c76\": container with ID starting with 6d618c4cef412e22402f92cbb9bba4fc0122b35c48d9dd47351e462a728d8c76 not found: ID does not exist" Oct 30 02:47:03.391002 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:03.390964 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9"] Oct 30 02:47:03.395043 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:03.395012 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-24fb97-openshift-default-7b8d9f6cdf-mljv9"] Oct 30 02:47:03.397614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:03.397579 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:47:03.619942 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:03.619847 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:03.726216 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:03.726179 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:04.178291 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:04.178213 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:04.539195 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:04.539109 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="820068d2-ba3f-4127-8f01-3b21c61acebd" path="/var/lib/kubelet/pods/820068d2-ba3f-4127-8f01-3b21c61acebd/volumes" Oct 30 02:47:04.589742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:04.589705 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:05.622909 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:05.622868 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:05.985885 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:05.985846 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:06.584139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:06.584084 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:07.755727 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:07.755683 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:07.864287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:07.864255 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:08.273036 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:08.272997 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:09.555906 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:09.555873 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:09.877947 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:09.877855 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:09.995882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:09.995835 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:10.106366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:10.106330 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:11.267255 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:11.267217 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:11.912416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:11.912373 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:12.257706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:12.257619 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:12.707364 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:12.707330 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:12.836768 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:12.836732 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:13.443462 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:13.443423 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:13.572386 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:13.572349 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:13.841169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:13.841066 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:14.351579 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:14.351539 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:15.356326 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:15.356264 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:16.066387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:16.066340 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:17.317354 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:17.317315 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:18.647933 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:18.647889 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:19.281429 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:19.281390 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:19.318154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:19.318115 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw"] Oct 30 02:47:19.329767 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:19.329729 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv"] Oct 30 02:47:19.336069 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:19.336031 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw"] Oct 30 02:47:19.344709 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:19.344671 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx"] Oct 30 02:47:19.433908 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:19.433869 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:20.209996 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:20.209930 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:20.495579 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:20.495469 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:20.617244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:20.617185 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:21.031108 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:21.031054 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:21.052653 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:21.052616 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x"] Oct 30 02:47:21.063903 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:21.063864 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-867b366a-2sk6x"] Oct 30 02:47:21.143260 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:21.143221 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:21.265235 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:21.265187 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:21.619571 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:21.619535 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:21.665788 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:21.665752 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl"] Oct 30 02:47:21.666016 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:21.665985 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" podUID="5220aca0-19d7-444c-b92b-da5361f306dc" containerName="istio-proxy" containerID="cri-o://4b68904613460380e5d659b652bfdf2853da1b1e6cac0b2703e7a0f4fa44269a" gracePeriod=30 Oct 30 02:47:21.762268 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:21.762231 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:22.193308 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:22.193274 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:22.539397 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:22.539315 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15a69048-bb38-4e8d-9aed-012cc287ca39" path="/var/lib/kubelet/pods/15a69048-bb38-4e8d-9aed-012cc287ca39/volumes" Oct 30 02:47:22.683279 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:22.683244 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:24.079937 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:24.079901 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:24.191824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:24.191788 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:24.307576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:24.307532 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:24.739662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:24.739621 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:25.122835 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:25.122744 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw"] Oct 30 02:47:25.123268 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:25.123044 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" podUID="d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" containerName="istio-proxy" containerID="cri-o://c9945b7fa8fd6570ff58163afae1392dcc251e12fa5c4a4ef0beeda9fe9c65d8" gracePeriod=30 Oct 30 02:47:25.175922 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:25.175889 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:25.228223 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:47:25.228180 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:47:26.094436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:26.094396 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:26.770261 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:26.770221 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:26.877430 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:26.877397 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:26.926924 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:26.926892 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:47:27.100560 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.100461 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-workload-socket\") pod \"5220aca0-19d7-444c-b92b-da5361f306dc\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " Oct 30 02:47:27.100560 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.100529 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-workload-certs\") pod \"5220aca0-19d7-444c-b92b-da5361f306dc\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " Oct 30 02:47:27.100824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.100562 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5220aca0-19d7-444c-b92b-da5361f306dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5220aca0-19d7-444c-b92b-da5361f306dc-istio-podinfo\") pod \"5220aca0-19d7-444c-b92b-da5361f306dc\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " Oct 30 02:47:27.100824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.100638 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5220aca0-19d7-444c-b92b-da5361f306dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5220aca0-19d7-444c-b92b-da5361f306dc-istiod-ca-cert\") pod \"5220aca0-19d7-444c-b92b-da5361f306dc\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " Oct 30 02:47:27.100824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.100668 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5220aca0-19d7-444c-b92b-da5361f306dc-kube-api-access-b95gv\" (UniqueName: \"kubernetes.io/projected/5220aca0-19d7-444c-b92b-da5361f306dc-kube-api-access-b95gv\") pod \"5220aca0-19d7-444c-b92b-da5361f306dc\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " Oct 30 02:47:27.100824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.100719 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-credential-socket\") pod \"5220aca0-19d7-444c-b92b-da5361f306dc\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " Oct 30 02:47:27.100824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.100752 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5220aca0-19d7-444c-b92b-da5361f306dc-istio-token\" (UniqueName: \"kubernetes.io/projected/5220aca0-19d7-444c-b92b-da5361f306dc-istio-token\") pod \"5220aca0-19d7-444c-b92b-da5361f306dc\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " Oct 30 02:47:27.100824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.100795 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-istio-envoy\") pod \"5220aca0-19d7-444c-b92b-da5361f306dc\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " Oct 30 02:47:27.101195 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.100831 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-workload-socket" pod "5220aca0-19d7-444c-b92b-da5361f306dc" (UID: "5220aca0-19d7-444c-b92b-da5361f306dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:27.101195 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.100859 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-istio-data\") pod \"5220aca0-19d7-444c-b92b-da5361f306dc\" (UID: \"5220aca0-19d7-444c-b92b-da5361f306dc\") " Oct 30 02:47:27.101195 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.100878 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-workload-certs" pod "5220aca0-19d7-444c-b92b-da5361f306dc" (UID: "5220aca0-19d7-444c-b92b-da5361f306dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:27.101195 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.101142 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5220aca0-19d7-444c-b92b-da5361f306dc-istiod-ca-cert" pod "5220aca0-19d7-444c-b92b-da5361f306dc" (UID: "5220aca0-19d7-444c-b92b-da5361f306dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:27.101422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.101235 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-credential-socket" pod "5220aca0-19d7-444c-b92b-da5361f306dc" (UID: "5220aca0-19d7-444c-b92b-da5361f306dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:27.101422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.101299 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-istio-data" pod "5220aca0-19d7-444c-b92b-da5361f306dc" (UID: "5220aca0-19d7-444c-b92b-da5361f306dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:27.101422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.101379 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:27.101422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.101394 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:27.101422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.101403 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5220aca0-19d7-444c-b92b-da5361f306dc-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:27.101422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.101411 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:27.103057 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.103030 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-istio-envoy" pod "5220aca0-19d7-444c-b92b-da5361f306dc" (UID: "5220aca0-19d7-444c-b92b-da5361f306dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:27.103249 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.103173 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5220aca0-19d7-444c-b92b-da5361f306dc-istio-podinfo" pod "5220aca0-19d7-444c-b92b-da5361f306dc" (UID: "5220aca0-19d7-444c-b92b-da5361f306dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:47:27.103404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.103388 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5220aca0-19d7-444c-b92b-da5361f306dc-istio-token" pod "5220aca0-19d7-444c-b92b-da5361f306dc" (UID: "5220aca0-19d7-444c-b92b-da5361f306dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:27.103531 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.103508 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5220aca0-19d7-444c-b92b-da5361f306dc-kube-api-access-b95gv" pod "5220aca0-19d7-444c-b92b-da5361f306dc" (UID: "5220aca0-19d7-444c-b92b-da5361f306dc"). InnerVolumeSpecName "kube-api-access-b95gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:27.188967 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.188928 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p"] Oct 30 02:47:27.202422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.202386 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5220aca0-19d7-444c-b92b-da5361f306dc-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:27.202422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.202417 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b95gv\" (UniqueName: \"kubernetes.io/projected/5220aca0-19d7-444c-b92b-da5361f306dc-kube-api-access-b95gv\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:27.202632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.202435 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5220aca0-19d7-444c-b92b-da5361f306dc-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:27.202632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.202448 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:27.202632 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.202459 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5220aca0-19d7-444c-b92b-da5361f306dc-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:27.281776 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.281739 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:27.476355 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.476313 5998 generic.go:356] "Generic (PLEG): container finished" podID="5220aca0-19d7-444c-b92b-da5361f306dc" containerID="4b68904613460380e5d659b652bfdf2853da1b1e6cac0b2703e7a0f4fa44269a" exitCode=0 Oct 30 02:47:27.476513 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.476377 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" Oct 30 02:47:27.476513 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.476397 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" event={"ID":"5220aca0-19d7-444c-b92b-da5361f306dc","Type":"ContainerDied","Data":"4b68904613460380e5d659b652bfdf2853da1b1e6cac0b2703e7a0f4fa44269a"} Oct 30 02:47:27.476513 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.476458 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl" event={"ID":"5220aca0-19d7-444c-b92b-da5361f306dc","Type":"ContainerDied","Data":"a51dd581415eaa3ec933979098c146949c07db59e13034b4a6ef871f346c0c4b"} Oct 30 02:47:27.476513 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.476483 5998 scope.go:122] "RemoveContainer" containerID="4b68904613460380e5d659b652bfdf2853da1b1e6cac0b2703e7a0f4fa44269a" Oct 30 02:47:27.487507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.487485 5998 scope.go:122] "RemoveContainer" containerID="178b62369ec547602268c4a39da502ed1f1a1d42b6ffddfc1a60baacbbbcf2a2" Oct 30 02:47:27.499675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.499650 5998 scope.go:122] "RemoveContainer" containerID="4b68904613460380e5d659b652bfdf2853da1b1e6cac0b2703e7a0f4fa44269a" Oct 30 02:47:27.499973 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:47:27.499951 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b68904613460380e5d659b652bfdf2853da1b1e6cac0b2703e7a0f4fa44269a\": container with ID starting with 4b68904613460380e5d659b652bfdf2853da1b1e6cac0b2703e7a0f4fa44269a not found: ID does not exist" containerID="4b68904613460380e5d659b652bfdf2853da1b1e6cac0b2703e7a0f4fa44269a" Oct 30 02:47:27.500034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.499984 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b68904613460380e5d659b652bfdf2853da1b1e6cac0b2703e7a0f4fa44269a"} err="failed to get container status \"4b68904613460380e5d659b652bfdf2853da1b1e6cac0b2703e7a0f4fa44269a\": rpc error: code = NotFound desc = could not find container \"4b68904613460380e5d659b652bfdf2853da1b1e6cac0b2703e7a0f4fa44269a\": container with ID starting with 4b68904613460380e5d659b652bfdf2853da1b1e6cac0b2703e7a0f4fa44269a not found: ID does not exist" Oct 30 02:47:27.500034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.500002 5998 scope.go:122] "RemoveContainer" containerID="178b62369ec547602268c4a39da502ed1f1a1d42b6ffddfc1a60baacbbbcf2a2" Oct 30 02:47:27.500300 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:47:27.500275 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"178b62369ec547602268c4a39da502ed1f1a1d42b6ffddfc1a60baacbbbcf2a2\": container with ID starting with 178b62369ec547602268c4a39da502ed1f1a1d42b6ffddfc1a60baacbbbcf2a2 not found: ID does not exist" containerID="178b62369ec547602268c4a39da502ed1f1a1d42b6ffddfc1a60baacbbbcf2a2" Oct 30 02:47:27.500408 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.500302 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"178b62369ec547602268c4a39da502ed1f1a1d42b6ffddfc1a60baacbbbcf2a2"} err="failed to get container status \"178b62369ec547602268c4a39da502ed1f1a1d42b6ffddfc1a60baacbbbcf2a2\": rpc error: code = NotFound desc = could not find container \"178b62369ec547602268c4a39da502ed1f1a1d42b6ffddfc1a60baacbbbcf2a2\": container with ID starting with 178b62369ec547602268c4a39da502ed1f1a1d42b6ffddfc1a60baacbbbcf2a2 not found: ID does not exist" Oct 30 02:47:27.525017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.524975 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl"] Oct 30 02:47:27.534727 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.534694 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9a1cd0-openshift-default-d569ddc65-qpvvl"] Oct 30 02:47:27.880355 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:27.880271 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:28.067749 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:28.067714 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:28.439616 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:28.439578 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:28.539434 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:28.539396 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5220aca0-19d7-444c-b92b-da5361f306dc" path="/var/lib/kubelet/pods/5220aca0-19d7-444c-b92b-da5361f306dc/volumes" Oct 30 02:47:29.335237 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:29.335195 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:47:29.495974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:29.495938 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:30.402283 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.402214 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 02:47:30.437193 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.437159 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-envoy\") pod \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " Oct 30 02:47:30.437391 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.437219 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-kube-api-access-m2l2z\" (UniqueName: \"kubernetes.io/projected/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-kube-api-access-m2l2z\") pod \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " Oct 30 02:47:30.437391 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.437266 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-data\") pod \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " Oct 30 02:47:30.437391 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.437303 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istiod-ca-cert\") pod \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " Oct 30 02:47:30.437391 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.437356 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-podinfo\") pod \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " Oct 30 02:47:30.437626 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.437416 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-token\" (UniqueName: \"kubernetes.io/projected/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-token\") pod \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " Oct 30 02:47:30.437626 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.437457 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-workload-certs\") pod \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " Oct 30 02:47:30.437626 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.437494 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-credential-socket\") pod \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " Oct 30 02:47:30.437626 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.437533 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-workload-socket\") pod \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\" (UID: \"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5\") " Oct 30 02:47:30.437915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.437881 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istiod-ca-cert" pod "d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" (UID: "d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:30.438167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.438079 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-workload-certs" pod "d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" (UID: "d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:30.438167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.438110 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-workload-socket" pod "d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" (UID: "d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:30.438422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.438016 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-credential-socket" pod "d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" (UID: "d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:30.438612 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.438564 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-data" pod "d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" (UID: "d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:30.440625 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.440598 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-token" pod "d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" (UID: "d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:30.440776 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.440746 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-podinfo" pod "d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" (UID: "d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:47:30.441484 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.441202 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-envoy" pod "d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" (UID: "d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:30.441635 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.441607 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-kube-api-access-m2l2z" pod "d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" (UID: "d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5"). InnerVolumeSpecName "kube-api-access-m2l2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:30.499154 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.499105 5998 generic.go:356] "Generic (PLEG): container finished" podID="d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" containerID="c9945b7fa8fd6570ff58163afae1392dcc251e12fa5c4a4ef0beeda9fe9c65d8" exitCode=0 Oct 30 02:47:30.499330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.499170 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" Oct 30 02:47:30.499330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.499189 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" event={"ID":"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5","Type":"ContainerDied","Data":"c9945b7fa8fd6570ff58163afae1392dcc251e12fa5c4a4ef0beeda9fe9c65d8"} Oct 30 02:47:30.499330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.499257 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw" event={"ID":"d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5","Type":"ContainerDied","Data":"891564830b21c6551cf88f67988a4d2a1ab047fee4d8071a3fd737dddc79c50d"} Oct 30 02:47:30.499330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.499290 5998 scope.go:122] "RemoveContainer" containerID="c9945b7fa8fd6570ff58163afae1392dcc251e12fa5c4a4ef0beeda9fe9c65d8" Oct 30 02:47:30.516908 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.516883 5998 scope.go:122] "RemoveContainer" containerID="c9945b7fa8fd6570ff58163afae1392dcc251e12fa5c4a4ef0beeda9fe9c65d8" Oct 30 02:47:30.517650 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:47:30.517614 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9945b7fa8fd6570ff58163afae1392dcc251e12fa5c4a4ef0beeda9fe9c65d8\": container with ID starting with c9945b7fa8fd6570ff58163afae1392dcc251e12fa5c4a4ef0beeda9fe9c65d8 not found: ID does not exist" containerID="c9945b7fa8fd6570ff58163afae1392dcc251e12fa5c4a4ef0beeda9fe9c65d8" Oct 30 02:47:30.517778 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.517663 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9945b7fa8fd6570ff58163afae1392dcc251e12fa5c4a4ef0beeda9fe9c65d8"} err="failed to get container status \"c9945b7fa8fd6570ff58163afae1392dcc251e12fa5c4a4ef0beeda9fe9c65d8\": rpc error: code = NotFound desc = could not find container \"c9945b7fa8fd6570ff58163afae1392dcc251e12fa5c4a4ef0beeda9fe9c65d8\": container with ID starting with c9945b7fa8fd6570ff58163afae1392dcc251e12fa5c4a4ef0beeda9fe9c65d8 not found: ID does not exist" Oct 30 02:47:30.538614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.538573 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:30.538614 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.538615 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2l2z\" (UniqueName: \"kubernetes.io/projected/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-kube-api-access-m2l2z\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:30.538871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.538632 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:30.538871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.538647 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:30.538871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.538660 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:30.538871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.538672 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:30.538871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.538684 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:30.538871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.538696 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:30.538871 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.538711 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:30.580687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.580648 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw"] Oct 30 02:47:30.591808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:30.591767 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-719029-openshift-default-75b56556b9-qvppw"] Oct 30 02:47:31.280755 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:31.280672 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:32.004646 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:32.004598 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:32.031148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:32.031116 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5"] Oct 30 02:47:32.041062 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:32.041035 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d07dacc7-fjjf5"] Oct 30 02:47:32.537978 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:32.537937 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5664b661-de95-4e5a-bffd-84a09455f58c" path="/var/lib/kubelet/pods/5664b661-de95-4e5a-bffd-84a09455f58c/volumes" Oct 30 02:47:32.538503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:32.538487 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" path="/var/lib/kubelet/pods/d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5/volumes" Oct 30 02:47:32.887602 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:32.887517 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:34.002882 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.002840 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:34.118185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.118144 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:34.350151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.350040 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" containerID="cri-o://831fde1f0a08333d72fc773b58fc3c3cd379a0b256722e9b5f7d93237d8cab19" gracePeriod=45 Oct 30 02:47:34.359638 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.359592 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" containerID="cri-o://761fec5f93f95a35af8f4bf8b7fcd8dfa04db9ddbbf7333db5dd63e13eb8765e" gracePeriod=45 Oct 30 02:47:34.363651 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.363615 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" containerID="cri-o://c093e6c10ab0574dc2a9a8192a1a6a199ca755800651847952260f2896895ade" gracePeriod=45 Oct 30 02:47:34.365373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.365339 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": EOF" Oct 30 02:47:34.368146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.367933 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" containerID="cri-o://ed3802a5221cffbad50df98efa8690453f3ac1ebd17d9c35915028ead3609c1d" gracePeriod=45 Oct 30 02:47:34.390800 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.390766 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": dial tcp 10.133.0.44:8000: connect: connection refused" Oct 30 02:47:34.529417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.529276 5998 generic.go:356] "Generic (PLEG): container finished" podID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerID="761fec5f93f95a35af8f4bf8b7fcd8dfa04db9ddbbf7333db5dd63e13eb8765e" exitCode=0 Oct 30 02:47:34.529417 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.529360 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" event={"ID":"bc2afe3c-bb27-460f-a7a0-abf3934daf2d","Type":"ContainerDied","Data":"761fec5f93f95a35af8f4bf8b7fcd8dfa04db9ddbbf7333db5dd63e13eb8765e"} Oct 30 02:47:34.535122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.535041 5998 generic.go:356] "Generic (PLEG): container finished" podID="bae44a76-84eb-475f-abea-de3828e1f033" containerID="ed3802a5221cffbad50df98efa8690453f3ac1ebd17d9c35915028ead3609c1d" exitCode=0 Oct 30 02:47:34.539521 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.539466 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" event={"ID":"bae44a76-84eb-475f-abea-de3828e1f033","Type":"ContainerDied","Data":"ed3802a5221cffbad50df98efa8690453f3ac1ebd17d9c35915028ead3609c1d"} Oct 30 02:47:34.540468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.540155 5998 generic.go:356] "Generic (PLEG): container finished" podID="4e218532-053c-41f2-8114-ab1385bbc759" containerID="831fde1f0a08333d72fc773b58fc3c3cd379a0b256722e9b5f7d93237d8cab19" exitCode=0 Oct 30 02:47:34.540468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.540233 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" event={"ID":"4e218532-053c-41f2-8114-ab1385bbc759","Type":"ContainerDied","Data":"831fde1f0a08333d72fc773b58fc3c3cd379a0b256722e9b5f7d93237d8cab19"} Oct 30 02:47:34.541997 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.541901 5998 generic.go:356] "Generic (PLEG): container finished" podID="63663089-b7d3-4565-a95a-3b06e913271b" containerID="c093e6c10ab0574dc2a9a8192a1a6a199ca755800651847952260f2896895ade" exitCode=0 Oct 30 02:47:34.541997 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.541956 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" event={"ID":"63663089-b7d3-4565-a95a-3b06e913271b","Type":"ContainerDied","Data":"c093e6c10ab0574dc2a9a8192a1a6a199ca755800651847952260f2896895ade"} Oct 30 02:47:34.659675 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.658161 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 02:47:34.678427 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.678398 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-tls-certs\" (UniqueName: \"kubernetes.io/secret/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-tls-certs\") pod \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " Oct 30 02:47:34.678636 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.678618 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-kube-api-access-mfbw7\" (UniqueName: \"kubernetes.io/projected/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-kube-api-access-mfbw7\") pod \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " Oct 30 02:47:34.678795 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.678782 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-home\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-home\") pod \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " Oct 30 02:47:34.678935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.678923 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-dshm\") pod \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " Oct 30 02:47:34.679073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.679060 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-model-cache\") pod \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " Oct 30 02:47:34.679219 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.679203 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-tmp-dir\") pod \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\" (UID: \"bc2afe3c-bb27-460f-a7a0-abf3934daf2d\") " Oct 30 02:47:34.679950 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.679927 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-tmp-dir" pod "bc2afe3c-bb27-460f-a7a0-abf3934daf2d" (UID: "bc2afe3c-bb27-460f-a7a0-abf3934daf2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:34.680307 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.680287 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-model-cache" pod "bc2afe3c-bb27-460f-a7a0-abf3934daf2d" (UID: "bc2afe3c-bb27-460f-a7a0-abf3934daf2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:34.680416 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.680311 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-home" pod "bc2afe3c-bb27-460f-a7a0-abf3934daf2d" (UID: "bc2afe3c-bb27-460f-a7a0-abf3934daf2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:34.681870 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.681840 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-kube-api-access-mfbw7" pod "bc2afe3c-bb27-460f-a7a0-abf3934daf2d" (UID: "bc2afe3c-bb27-460f-a7a0-abf3934daf2d"). InnerVolumeSpecName "kube-api-access-mfbw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:34.682138 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.682103 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-tls-certs" pod "bc2afe3c-bb27-460f-a7a0-abf3934daf2d" (UID: "bc2afe3c-bb27-460f-a7a0-abf3934daf2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:34.682947 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.682889 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-dshm" pod "bc2afe3c-bb27-460f-a7a0-abf3934daf2d" (UID: "bc2afe3c-bb27-460f-a7a0-abf3934daf2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:34.688798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.688773 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 02:47:34.691610 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.691587 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 02:47:34.696289 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.696269 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 02:47:34.780633 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.780592 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-tmp-dir\") pod \"4e218532-053c-41f2-8114-ab1385bbc759\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " Oct 30 02:47:34.781185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781147 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-dshm\") pod \"4e218532-053c-41f2-8114-ab1385bbc759\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " Oct 30 02:47:34.781352 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781073 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-tmp-dir" pod "4e218532-053c-41f2-8114-ab1385bbc759" (UID: "4e218532-053c-41f2-8114-ab1385bbc759"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:34.781352 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781208 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-model-cache\") pod \"4e218532-053c-41f2-8114-ab1385bbc759\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " Oct 30 02:47:34.781352 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781242 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-home\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-home\") pod \"4e218532-053c-41f2-8114-ab1385bbc759\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " Oct 30 02:47:34.781352 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781286 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63663089-b7d3-4565-a95a-3b06e913271b-tls-certs\" (UniqueName: \"kubernetes.io/secret/63663089-b7d3-4565-a95a-3b06e913271b-tls-certs\") pod \"63663089-b7d3-4565-a95a-3b06e913271b\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " Oct 30 02:47:34.781352 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781326 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-home\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-home\") pod \"bae44a76-84eb-475f-abea-de3828e1f033\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " Oct 30 02:47:34.781662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781367 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63663089-b7d3-4565-a95a-3b06e913271b-kube-api-access-m68z7\" (UniqueName: \"kubernetes.io/projected/63663089-b7d3-4565-a95a-3b06e913271b-kube-api-access-m68z7\") pod \"63663089-b7d3-4565-a95a-3b06e913271b\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " Oct 30 02:47:34.781662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781400 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e218532-053c-41f2-8114-ab1385bbc759-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e218532-053c-41f2-8114-ab1385bbc759-tls-certs\") pod \"4e218532-053c-41f2-8114-ab1385bbc759\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " Oct 30 02:47:34.781662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781459 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bae44a76-84eb-475f-abea-de3828e1f033-tls-certs\" (UniqueName: \"kubernetes.io/secret/bae44a76-84eb-475f-abea-de3828e1f033-tls-certs\") pod \"bae44a76-84eb-475f-abea-de3828e1f033\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " Oct 30 02:47:34.781662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781472 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-model-cache" pod "4e218532-053c-41f2-8114-ab1385bbc759" (UID: "4e218532-053c-41f2-8114-ab1385bbc759"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:34.781662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781489 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-dshm\") pod \"bae44a76-84eb-475f-abea-de3828e1f033\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " Oct 30 02:47:34.781662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781538 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-model-cache\") pod \"63663089-b7d3-4565-a95a-3b06e913271b\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " Oct 30 02:47:34.781662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781553 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-home" pod "4e218532-053c-41f2-8114-ab1385bbc759" (UID: "4e218532-053c-41f2-8114-ab1385bbc759"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:34.781662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781568 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e218532-053c-41f2-8114-ab1385bbc759-kube-api-access-b9ttz\" (UniqueName: \"kubernetes.io/projected/4e218532-053c-41f2-8114-ab1385bbc759-kube-api-access-b9ttz\") pod \"4e218532-053c-41f2-8114-ab1385bbc759\" (UID: \"4e218532-053c-41f2-8114-ab1385bbc759\") " Oct 30 02:47:34.781662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781623 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-tmp-dir\") pod \"63663089-b7d3-4565-a95a-3b06e913271b\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " Oct 30 02:47:34.782208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781677 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-home\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-home\") pod \"63663089-b7d3-4565-a95a-3b06e913271b\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " Oct 30 02:47:34.782208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781716 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae44a76-84eb-475f-abea-de3828e1f033-kube-api-access-w4lk6\" (UniqueName: \"kubernetes.io/projected/bae44a76-84eb-475f-abea-de3828e1f033-kube-api-access-w4lk6\") pod \"bae44a76-84eb-475f-abea-de3828e1f033\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " Oct 30 02:47:34.782208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781800 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-model-cache\") pod \"bae44a76-84eb-475f-abea-de3828e1f033\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " Oct 30 02:47:34.782208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781829 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-dshm\") pod \"63663089-b7d3-4565-a95a-3b06e913271b\" (UID: \"63663089-b7d3-4565-a95a-3b06e913271b\") " Oct 30 02:47:34.782208 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.781866 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-tmp-dir\") pod \"bae44a76-84eb-475f-abea-de3828e1f033\" (UID: \"bae44a76-84eb-475f-abea-de3828e1f033\") " Oct 30 02:47:34.782633 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.782605 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.782729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.782638 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.782729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.782653 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.782729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.782666 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.782729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.782679 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.782729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.782691 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.782729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.782705 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.782729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.782717 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.782729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.782731 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfbw7\" (UniqueName: \"kubernetes.io/projected/bc2afe3c-bb27-460f-a7a0-abf3934daf2d-kube-api-access-mfbw7\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.785003 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.783286 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-home" pod "63663089-b7d3-4565-a95a-3b06e913271b" (UID: "63663089-b7d3-4565-a95a-3b06e913271b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:34.785003 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.783598 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-tmp-dir" pod "63663089-b7d3-4565-a95a-3b06e913271b" (UID: "63663089-b7d3-4565-a95a-3b06e913271b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:34.785003 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.783829 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-home" pod "bae44a76-84eb-475f-abea-de3828e1f033" (UID: "bae44a76-84eb-475f-abea-de3828e1f033"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:34.785003 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.784349 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-tmp-dir" pod "bae44a76-84eb-475f-abea-de3828e1f033" (UID: "bae44a76-84eb-475f-abea-de3828e1f033"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:34.785003 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.784755 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-model-cache" pod "bae44a76-84eb-475f-abea-de3828e1f033" (UID: "bae44a76-84eb-475f-abea-de3828e1f033"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:34.787226 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.787199 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-model-cache" pod "63663089-b7d3-4565-a95a-3b06e913271b" (UID: "63663089-b7d3-4565-a95a-3b06e913271b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:34.788714 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.788692 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bae44a76-84eb-475f-abea-de3828e1f033-tls-certs" pod "bae44a76-84eb-475f-abea-de3828e1f033" (UID: "bae44a76-84eb-475f-abea-de3828e1f033"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:34.788714 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.788704 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-dshm" pod "63663089-b7d3-4565-a95a-3b06e913271b" (UID: "63663089-b7d3-4565-a95a-3b06e913271b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:34.788903 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.788711 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63663089-b7d3-4565-a95a-3b06e913271b-tls-certs" pod "63663089-b7d3-4565-a95a-3b06e913271b" (UID: "63663089-b7d3-4565-a95a-3b06e913271b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:34.788903 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.788843 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-dshm" pod "4e218532-053c-41f2-8114-ab1385bbc759" (UID: "4e218532-053c-41f2-8114-ab1385bbc759"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:34.788903 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.788843 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae44a76-84eb-475f-abea-de3828e1f033-kube-api-access-w4lk6" pod "bae44a76-84eb-475f-abea-de3828e1f033" (UID: "bae44a76-84eb-475f-abea-de3828e1f033"). InnerVolumeSpecName "kube-api-access-w4lk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:34.788903 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.788894 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63663089-b7d3-4565-a95a-3b06e913271b-kube-api-access-m68z7" pod "63663089-b7d3-4565-a95a-3b06e913271b" (UID: "63663089-b7d3-4565-a95a-3b06e913271b"). InnerVolumeSpecName "kube-api-access-m68z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:34.789576 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.789553 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-dshm" pod "bae44a76-84eb-475f-abea-de3828e1f033" (UID: "bae44a76-84eb-475f-abea-de3828e1f033"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:34.790422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.790394 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e218532-053c-41f2-8114-ab1385bbc759-tls-certs" pod "4e218532-053c-41f2-8114-ab1385bbc759" (UID: "4e218532-053c-41f2-8114-ab1385bbc759"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:34.790622 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.790600 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e218532-053c-41f2-8114-ab1385bbc759-kube-api-access-b9ttz" pod "4e218532-053c-41f2-8114-ab1385bbc759" (UID: "4e218532-053c-41f2-8114-ab1385bbc759"). InnerVolumeSpecName "kube-api-access-b9ttz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:34.884118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.884028 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e218532-053c-41f2-8114-ab1385bbc759-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.884118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.884060 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63663089-b7d3-4565-a95a-3b06e913271b-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.884118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.884070 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.884118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.884079 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m68z7\" (UniqueName: \"kubernetes.io/projected/63663089-b7d3-4565-a95a-3b06e913271b-kube-api-access-m68z7\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.884118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.884116 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e218532-053c-41f2-8114-ab1385bbc759-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.884407 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.884126 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bae44a76-84eb-475f-abea-de3828e1f033-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.884407 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.884133 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.884407 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.884143 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.884407 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.884154 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9ttz\" (UniqueName: \"kubernetes.io/projected/4e218532-053c-41f2-8114-ab1385bbc759-kube-api-access-b9ttz\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.884407 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.884165 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.884407 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.884174 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.884407 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.884182 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4lk6\" (UniqueName: \"kubernetes.io/projected/bae44a76-84eb-475f-abea-de3828e1f033-kube-api-access-w4lk6\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.884407 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.884190 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.884407 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.884198 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63663089-b7d3-4565-a95a-3b06e913271b-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:34.884407 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:34.884205 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bae44a76-84eb-475f-abea-de3828e1f033-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:35.065819 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.065787 5998 scope.go:122] "RemoveContainer" containerID="831fde1f0a08333d72fc773b58fc3c3cd379a0b256722e9b5f7d93237d8cab19" Oct 30 02:47:35.075649 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.075624 5998 scope.go:122] "RemoveContainer" containerID="ed3802a5221cffbad50df98efa8690453f3ac1ebd17d9c35915028ead3609c1d" Oct 30 02:47:35.084049 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.084024 5998 scope.go:122] "RemoveContainer" containerID="c093e6c10ab0574dc2a9a8192a1a6a199ca755800651847952260f2896895ade" Oct 30 02:47:35.092144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.092120 5998 scope.go:122] "RemoveContainer" containerID="761fec5f93f95a35af8f4bf8b7fcd8dfa04db9ddbbf7333db5dd63e13eb8765e" Oct 30 02:47:35.101277 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.101250 5998 scope.go:122] "RemoveContainer" containerID="72b6955641d324c3848a8fbd1ca29f7c9eecae83bcdb37b7a69e987328391d20" Oct 30 02:47:35.547141 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.547077 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" event={"ID":"bc2afe3c-bb27-460f-a7a0-abf3934daf2d","Type":"ContainerDied","Data":"8c320e1ef556504e3936db0d404d2ce1646f18bb8b39e3c4f1adccf6d5d0ab96"} Oct 30 02:47:35.547351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.547159 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" event={"ID":"bae44a76-84eb-475f-abea-de3828e1f033","Type":"ContainerDied","Data":"107db43a7c1776e4facbda5322416f94c5fe03bf35ce09ac31e38e1713dd4808"} Oct 30 02:47:35.547351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.547183 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv" Oct 30 02:47:35.547351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.547199 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" Oct 30 02:47:35.547351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.547192 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx" Oct 30 02:47:35.547351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.547220 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" Oct 30 02:47:35.547351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.547185 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw" event={"ID":"4e218532-053c-41f2-8114-ab1385bbc759","Type":"ContainerDied","Data":"28099f2ba918810f53aba9b21a8439a976e417ff5e932bde6e586e085c9ef6d6"} Oct 30 02:47:35.547351 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.547288 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw" event={"ID":"63663089-b7d3-4565-a95a-3b06e913271b","Type":"ContainerDied","Data":"0de8f9dbfd374e39c41e62fd43e1aea599a8518b9335bf413bdd3b7b2cce8bdd"} Oct 30 02:47:35.700324 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.700284 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx"] Oct 30 02:47:35.706606 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.706575 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/simulator-w3-78d5ff-kserve-9c499455d-vkgtx"] Oct 30 02:47:35.737076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.737041 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv"] Oct 30 02:47:35.747162 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.747120 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/distinct-w3-78d5ff-kserve-5fd9fbdbdb-zk4sv"] Oct 30 02:47:35.775459 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.775421 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw"] Oct 30 02:47:35.778454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.778428 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:35.780079 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.780056 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/distinct-2-w3-78d5ff-kserve-6668c844f7-jh4cw"] Oct 30 02:47:35.837303 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.837217 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw"] Oct 30 02:47:35.844999 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:35.844961 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-78d5ff/premium-w3-78d5ff-kserve-7b944bddc9-8fnfw"] Oct 30 02:47:36.171327 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:36.171288 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:36.539698 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:36.539618 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e218532-053c-41f2-8114-ab1385bbc759" path="/var/lib/kubelet/pods/4e218532-053c-41f2-8114-ab1385bbc759/volumes" Oct 30 02:47:36.540068 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:36.540054 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63663089-b7d3-4565-a95a-3b06e913271b" path="/var/lib/kubelet/pods/63663089-b7d3-4565-a95a-3b06e913271b/volumes" Oct 30 02:47:36.540472 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:36.540459 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae44a76-84eb-475f-abea-de3828e1f033" path="/var/lib/kubelet/pods/bae44a76-84eb-475f-abea-de3828e1f033/volumes" Oct 30 02:47:36.541138 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:36.541123 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" path="/var/lib/kubelet/pods/bc2afe3c-bb27-460f-a7a0-abf3934daf2d/volumes" Oct 30 02:47:38.178858 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:38.178814 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:38.457532 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:38.457387 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:38.567455 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:38.567413 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:39.613183 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:39.613145 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:40.932682 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:40.932635 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:41.769818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:41.769681 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:42.035366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.035277 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv"] Oct 30 02:47:42.043850 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.043817 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157dcde6-sztvv"] Oct 30 02:47:42.210282 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.210241 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" containerID="cri-o://fda30aa28d3fe47f53a954d9d2fb10c24d79501d5445b90b793786157ad4f2e1" gracePeriod=45 Oct 30 02:47:42.216061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.216022 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": EOF" Oct 30 02:47:42.468761 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.468728 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 02:47:42.539073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.539034 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b3112b-1f2d-480f-bf87-7a5fe060f999" path="/var/lib/kubelet/pods/a8b3112b-1f2d-480f-bf87-7a5fe060f999/volumes" Oct 30 02:47:42.560180 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.560150 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-tmp-dir\") pod \"a7c758e0-df24-4751-9751-24d3050bdcbb\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " Oct 30 02:47:42.560387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.560203 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c758e0-df24-4751-9751-24d3050bdcbb-kube-api-access-spntn\" (UniqueName: \"kubernetes.io/projected/a7c758e0-df24-4751-9751-24d3050bdcbb-kube-api-access-spntn\") pod \"a7c758e0-df24-4751-9751-24d3050bdcbb\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " Oct 30 02:47:42.560387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.560325 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-model-cache\") pod \"a7c758e0-df24-4751-9751-24d3050bdcbb\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " Oct 30 02:47:42.560387 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.560367 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-dshm\") pod \"a7c758e0-df24-4751-9751-24d3050bdcbb\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " Oct 30 02:47:42.560575 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.560403 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7c758e0-df24-4751-9751-24d3050bdcbb-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7c758e0-df24-4751-9751-24d3050bdcbb-tls-certs\") pod \"a7c758e0-df24-4751-9751-24d3050bdcbb\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " Oct 30 02:47:42.560575 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.560432 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-home\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-home\") pod \"a7c758e0-df24-4751-9751-24d3050bdcbb\" (UID: \"a7c758e0-df24-4751-9751-24d3050bdcbb\") " Oct 30 02:47:42.560575 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.560467 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-tmp-dir" pod "a7c758e0-df24-4751-9751-24d3050bdcbb" (UID: "a7c758e0-df24-4751-9751-24d3050bdcbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:42.560734 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.560651 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-model-cache" pod "a7c758e0-df24-4751-9751-24d3050bdcbb" (UID: "a7c758e0-df24-4751-9751-24d3050bdcbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:42.560827 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.560806 5998 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-model-cache\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.560885 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.560841 5998 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-tmp-dir\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.560885 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.560801 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-home" pod "a7c758e0-df24-4751-9751-24d3050bdcbb" (UID: "a7c758e0-df24-4751-9751-24d3050bdcbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:42.563080 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.563049 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c758e0-df24-4751-9751-24d3050bdcbb-kube-api-access-spntn" pod "a7c758e0-df24-4751-9751-24d3050bdcbb" (UID: "a7c758e0-df24-4751-9751-24d3050bdcbb"). InnerVolumeSpecName "kube-api-access-spntn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:42.563305 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.563280 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-dshm" pod "a7c758e0-df24-4751-9751-24d3050bdcbb" (UID: "a7c758e0-df24-4751-9751-24d3050bdcbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:42.563579 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.563550 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c758e0-df24-4751-9751-24d3050bdcbb-tls-certs" pod "a7c758e0-df24-4751-9751-24d3050bdcbb" (UID: "a7c758e0-df24-4751-9751-24d3050bdcbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:47:42.591029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.590943 5998 generic.go:356] "Generic (PLEG): container finished" podID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerID="fda30aa28d3fe47f53a954d9d2fb10c24d79501d5445b90b793786157ad4f2e1" exitCode=0 Oct 30 02:47:42.591029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.590989 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" Oct 30 02:47:42.591287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.591022 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" event={"ID":"a7c758e0-df24-4751-9751-24d3050bdcbb","Type":"ContainerDied","Data":"fda30aa28d3fe47f53a954d9d2fb10c24d79501d5445b90b793786157ad4f2e1"} Oct 30 02:47:42.591287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.591074 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p" event={"ID":"a7c758e0-df24-4751-9751-24d3050bdcbb","Type":"ContainerDied","Data":"7247185310bb83e1b93f411c844e647e34e1531d26f4d250c59df7ea1b55a6fc"} Oct 30 02:47:42.591287 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.591118 5998 scope.go:122] "RemoveContainer" containerID="fda30aa28d3fe47f53a954d9d2fb10c24d79501d5445b90b793786157ad4f2e1" Oct 30 02:47:42.603228 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.603198 5998 scope.go:122] "RemoveContainer" containerID="fda30aa28d3fe47f53a954d9d2fb10c24d79501d5445b90b793786157ad4f2e1" Oct 30 02:47:42.603568 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:47:42.603547 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fda30aa28d3fe47f53a954d9d2fb10c24d79501d5445b90b793786157ad4f2e1\": container with ID starting with fda30aa28d3fe47f53a954d9d2fb10c24d79501d5445b90b793786157ad4f2e1 not found: ID does not exist" containerID="fda30aa28d3fe47f53a954d9d2fb10c24d79501d5445b90b793786157ad4f2e1" Oct 30 02:47:42.603659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.603577 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fda30aa28d3fe47f53a954d9d2fb10c24d79501d5445b90b793786157ad4f2e1"} err="failed to get container status \"fda30aa28d3fe47f53a954d9d2fb10c24d79501d5445b90b793786157ad4f2e1\": rpc error: code = NotFound desc = could not find container \"fda30aa28d3fe47f53a954d9d2fb10c24d79501d5445b90b793786157ad4f2e1\": container with ID starting with fda30aa28d3fe47f53a954d9d2fb10c24d79501d5445b90b793786157ad4f2e1 not found: ID does not exist" Oct 30 02:47:42.649315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.649281 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p"] Oct 30 02:47:42.651528 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.651496 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c40a89/simulator-w0-c40a89-kserve-6db79f49b6-xq52p"] Oct 30 02:47:42.661428 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.661395 5998 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-home\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.661428 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.661429 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spntn\" (UniqueName: \"kubernetes.io/projected/a7c758e0-df24-4751-9751-24d3050bdcbb-kube-api-access-spntn\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.661611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.661443 5998 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7c758e0-df24-4751-9751-24d3050bdcbb-dshm\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:42.661611 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:42.661457 5998 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7c758e0-df24-4751-9751-24d3050bdcbb-tls-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:44.128975 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:44.128938 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:44.252867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:44.252833 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:44.540389 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:44.540353 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" path="/var/lib/kubelet/pods/a7c758e0-df24-4751-9751-24d3050bdcbb/volumes" Oct 30 02:47:44.915483 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:44.915445 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:45.027781 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:45.027737 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:45.323011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:45.322926 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx"] Oct 30 02:47:45.323489 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:45.323294 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" podUID="ff075586-a6a8-4eed-819d-19c55abfa057" containerName="istio-proxy" containerID="cri-o://0168693482f2b152302e51f5be14af7f8b236c6d8cf95c3a4e77d193457074bd" gracePeriod=30 Oct 30 02:47:45.346012 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:45.345972 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:46.815424 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:46.815382 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:46.963945 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:46.963904 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:50.081498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.081463 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:47:50.578061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.578034 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 02:47:50.637427 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.637347 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff075586-a6a8-4eed-819d-19c55abfa057-kube-api-access-vlb9w\" (UniqueName: \"kubernetes.io/projected/ff075586-a6a8-4eed-819d-19c55abfa057-kube-api-access-vlb9w\") pod \"ff075586-a6a8-4eed-819d-19c55abfa057\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " Oct 30 02:47:50.637427 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.637408 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-workload-certs\") pod \"ff075586-a6a8-4eed-819d-19c55abfa057\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " Oct 30 02:47:50.637680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.637473 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff075586-a6a8-4eed-819d-19c55abfa057-istio-token\" (UniqueName: \"kubernetes.io/projected/ff075586-a6a8-4eed-819d-19c55abfa057-istio-token\") pod \"ff075586-a6a8-4eed-819d-19c55abfa057\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " Oct 30 02:47:50.637680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.637505 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff075586-a6a8-4eed-819d-19c55abfa057-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff075586-a6a8-4eed-819d-19c55abfa057-istio-podinfo\") pod \"ff075586-a6a8-4eed-819d-19c55abfa057\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " Oct 30 02:47:50.637680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.637534 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff075586-a6a8-4eed-819d-19c55abfa057-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff075586-a6a8-4eed-819d-19c55abfa057-istiod-ca-cert\") pod \"ff075586-a6a8-4eed-819d-19c55abfa057\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " Oct 30 02:47:50.637680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.637557 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-credential-socket\") pod \"ff075586-a6a8-4eed-819d-19c55abfa057\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " Oct 30 02:47:50.637680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.637595 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-workload-socket\") pod \"ff075586-a6a8-4eed-819d-19c55abfa057\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " Oct 30 02:47:50.637680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.637629 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-istio-data\") pod \"ff075586-a6a8-4eed-819d-19c55abfa057\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " Oct 30 02:47:50.638027 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.637693 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-istio-envoy\") pod \"ff075586-a6a8-4eed-819d-19c55abfa057\" (UID: \"ff075586-a6a8-4eed-819d-19c55abfa057\") " Oct 30 02:47:50.638027 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.637940 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-credential-socket" pod "ff075586-a6a8-4eed-819d-19c55abfa057" (UID: "ff075586-a6a8-4eed-819d-19c55abfa057"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:50.638185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.638164 5998 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-credential-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:50.638779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.638398 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-workload-certs" pod "ff075586-a6a8-4eed-819d-19c55abfa057" (UID: "ff075586-a6a8-4eed-819d-19c55abfa057"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:50.638779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.638504 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff075586-a6a8-4eed-819d-19c55abfa057-istiod-ca-cert" pod "ff075586-a6a8-4eed-819d-19c55abfa057" (UID: "ff075586-a6a8-4eed-819d-19c55abfa057"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:47:50.638779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.638684 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-istio-data" pod "ff075586-a6a8-4eed-819d-19c55abfa057" (UID: "ff075586-a6a8-4eed-819d-19c55abfa057"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:50.638779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.638749 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-workload-socket" pod "ff075586-a6a8-4eed-819d-19c55abfa057" (UID: "ff075586-a6a8-4eed-819d-19c55abfa057"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:50.640480 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.640422 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-istio-envoy" pod "ff075586-a6a8-4eed-819d-19c55abfa057" (UID: "ff075586-a6a8-4eed-819d-19c55abfa057"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:47:50.640941 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.640916 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff075586-a6a8-4eed-819d-19c55abfa057-kube-api-access-vlb9w" pod "ff075586-a6a8-4eed-819d-19c55abfa057" (UID: "ff075586-a6a8-4eed-819d-19c55abfa057"). InnerVolumeSpecName "kube-api-access-vlb9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:50.641065 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.640946 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff075586-a6a8-4eed-819d-19c55abfa057-istio-token" pod "ff075586-a6a8-4eed-819d-19c55abfa057" (UID: "ff075586-a6a8-4eed-819d-19c55abfa057"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:47:50.641916 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.641878 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff075586-a6a8-4eed-819d-19c55abfa057-istio-podinfo" pod "ff075586-a6a8-4eed-819d-19c55abfa057" (UID: "ff075586-a6a8-4eed-819d-19c55abfa057"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:47:50.655742 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.655716 5998 generic.go:356] "Generic (PLEG): container finished" podID="ff075586-a6a8-4eed-819d-19c55abfa057" containerID="0168693482f2b152302e51f5be14af7f8b236c6d8cf95c3a4e77d193457074bd" exitCode=0 Oct 30 02:47:50.655874 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.655783 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" Oct 30 02:47:50.655874 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.655800 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" event={"ID":"ff075586-a6a8-4eed-819d-19c55abfa057","Type":"ContainerDied","Data":"0168693482f2b152302e51f5be14af7f8b236c6d8cf95c3a4e77d193457074bd"} Oct 30 02:47:50.655874 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.655858 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx" event={"ID":"ff075586-a6a8-4eed-819d-19c55abfa057","Type":"ContainerDied","Data":"ad498602b214adfd921eb56b24677ffb51eb15416e91ae584476c40429433b11"} Oct 30 02:47:50.656055 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.655885 5998 scope.go:122] "RemoveContainer" containerID="0168693482f2b152302e51f5be14af7f8b236c6d8cf95c3a4e77d193457074bd" Oct 30 02:47:50.667925 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.667889 5998 scope.go:122] "RemoveContainer" containerID="0168693482f2b152302e51f5be14af7f8b236c6d8cf95c3a4e77d193457074bd" Oct 30 02:47:50.668248 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:47:50.668225 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0168693482f2b152302e51f5be14af7f8b236c6d8cf95c3a4e77d193457074bd\": container with ID starting with 0168693482f2b152302e51f5be14af7f8b236c6d8cf95c3a4e77d193457074bd not found: ID does not exist" containerID="0168693482f2b152302e51f5be14af7f8b236c6d8cf95c3a4e77d193457074bd" Oct 30 02:47:50.668341 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.668260 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0168693482f2b152302e51f5be14af7f8b236c6d8cf95c3a4e77d193457074bd"} err="failed to get container status \"0168693482f2b152302e51f5be14af7f8b236c6d8cf95c3a4e77d193457074bd\": rpc error: code = NotFound desc = could not find container \"0168693482f2b152302e51f5be14af7f8b236c6d8cf95c3a4e77d193457074bd\": container with ID starting with 0168693482f2b152302e51f5be14af7f8b236c6d8cf95c3a4e77d193457074bd not found: ID does not exist" Oct 30 02:47:50.708599 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.708557 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx"] Oct 30 02:47:50.708801 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.708618 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-78d5ff-openshift-default-5c9bf5c56c-qrlzx"] Oct 30 02:47:50.738688 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.738648 5998 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff075586-a6a8-4eed-819d-19c55abfa057-istiod-ca-cert\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:50.738688 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.738683 5998 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-workload-socket\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:50.738688 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.738691 5998 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-istio-data\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:50.738688 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.738699 5998 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-istio-envoy\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:50.738971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.738708 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlb9w\" (UniqueName: \"kubernetes.io/projected/ff075586-a6a8-4eed-819d-19c55abfa057-kube-api-access-vlb9w\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:50.738971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.738718 5998 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff075586-a6a8-4eed-819d-19c55abfa057-workload-certs\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:50.738971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.738727 5998 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff075586-a6a8-4eed-819d-19c55abfa057-istio-token\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:50.738971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:50.738735 5998 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff075586-a6a8-4eed-819d-19c55abfa057-istio-podinfo\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:47:51.811513 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:51.811469 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:51.922434 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:51.922398 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:52.486973 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:52.486928 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:52.538613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:52.538571 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff075586-a6a8-4eed-819d-19c55abfa057" path="/var/lib/kubelet/pods/ff075586-a6a8-4eed-819d-19c55abfa057/volumes" Oct 30 02:47:53.004866 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:53.004829 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:53.032917 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:53.032877 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54"] Oct 30 02:47:53.043123 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:53.043078 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e3c8990-tnr54"] Oct 30 02:47:54.501948 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:54.501915 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-z4b6g_db689d73-bcb1-41fb-8507-84109c764878/manager/0.log" Oct 30 02:47:54.537615 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:54.537569 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eb180c8-0839-4632-8fe3-613d4e5afb82" path="/var/lib/kubelet/pods/7eb180c8-0839-4632-8fe3-613d4e5afb82/volumes" Oct 30 02:47:55.128749 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:55.128708 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-rjj9l_9b4e4651-47b0-4440-b8a1-87d7fdb96f46/manager/0.log" Oct 30 02:47:55.216322 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:47:55.216279 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:47:55.391545 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:55.391457 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-p2m8w_db342fa8-f998-43b9-8e0a-5da7612cdee2/limitador/0.log" Oct 30 02:47:56.592444 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:56.592406 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:56.698320 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:56.698238 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:47:56.791217 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:56.791189 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-25r9k_7994cb01-5d42-4871-ae17-d9622123f971/manager/0.log" Oct 30 02:47:57.003984 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:57.003949 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zr7w4_114ffe72-9ce9-4eed-baea-d3667e969148/manager/0.log" Oct 30 02:47:57.236388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:57.236355 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-wptrw_1a3c5537-f83e-412e-a917-27fd46cfe258/manager/0.log" Oct 30 02:47:58.016005 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:58.015971 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9fjjh_712fc2f5-d537-4e96-9801-963522329de7/server/0.log" Oct 30 02:47:58.204055 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:58.204021 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kpkmz_04727d53-1a8f-4f08-9930-acfb77f05eb4/manager/0.log" Oct 30 02:47:58.560848 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:47:58.560813 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:02.944173 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:02.944071 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:03.470076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:03.470033 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:03.730836 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:03.730738 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:03.903766 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:03.903729 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:04.014074 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:04.013987 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:06.037563 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:06.031882 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f"] Oct 30 02:48:06.043948 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:06.043910 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22df452a-v577f"] Oct 30 02:48:06.179538 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:06.179504 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:06.286698 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:06.286659 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:06.538976 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:06.538941 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e8ed90-b3fb-4374-9380-733684ba2cb9" path="/var/lib/kubelet/pods/c5e8ed90-b3fb-4374-9380-733684ba2cb9/volumes" Oct 30 02:48:06.924832 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:06.924796 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:07.106315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:07.106269 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:07.653441 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:07.653410 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:07.860331 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:07.860298 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:07.981168 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:07.981125 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:08.098798 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:08.098759 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:09.399160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:09.399123 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:10.874887 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:10.874850 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:48:10.906865 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:10.906824 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:12.615584 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:12.615547 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:13.958005 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:13.957960 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:14.167147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:14.167085 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:15.678901 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:15.678860 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:17.115921 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:17.115875 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx"] Oct 30 02:48:17.134249 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:17.134217 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dfa458b-k58xx"] Oct 30 02:48:17.287350 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:17.287314 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:18.537560 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:18.537518 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87bd750b-f9a3-49b0-a1f5-fbbf12033834" path="/var/lib/kubelet/pods/87bd750b-f9a3-49b0-a1f5-fbbf12033834/volumes" Oct 30 02:48:18.866315 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:18.866225 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:20.567785 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:20.567750 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:22.125928 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:22.125886 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:23.779988 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:23.779949 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:25.217047 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:48:25.217004 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:48:25.274061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:25.274024 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:26.642596 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:26.642563 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:26.758083 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:26.758049 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:26.866381 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:26.866335 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:28.379337 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:28.379279 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:29.890858 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:29.890820 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:31.377865 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:31.377775 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:32.560365 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:32.560331 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:32.669789 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:32.669753 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:32.779498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:32.779458 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:33.736251 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:33.736216 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:33.861521 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:33.861482 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:35.172974 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:35.172935 5998 scope.go:122] "RemoveContainer" containerID="c2e7ca79a0e3ebc7ebbb2bd85f615d30b45ecdc9458a738b68acac6f95e8b39f" Oct 30 02:48:35.181975 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:35.181949 5998 scope.go:122] "RemoveContainer" containerID="6e93360f6ca39c0250c5a671d2e23c545e2458912927e602c963bc8a7ae78669" Oct 30 02:48:35.190194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:35.190173 5998 scope.go:122] "RemoveContainer" containerID="af3b9dce154db9e5d8c45d9592c427e087e5391a2f8148a3162d6c1be89cb323" Oct 30 02:48:35.685463 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:35.685419 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:48:38.344564 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:38.344524 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:41.907033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:48:41.906992 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:48:55.215987 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:48:55.215944 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:49:02.515244 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:02.515203 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:49:02.620030 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:02.619990 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:49:06.102585 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.102547 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd"] Oct 30 02:49:06.103073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.102948 5998 cpu_manager.go:414] "containerMap: removing container" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" Oct 30 02:49:06.103073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.102961 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" Oct 30 02:49:06.103073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.102980 5998 cpu_manager.go:414] "containerMap: removing container" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" Oct 30 02:49:06.103073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.102988 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" Oct 30 02:49:06.103073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.102997 5998 cpu_manager.go:414] "containerMap: removing container" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" Oct 30 02:49:06.103073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103002 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" Oct 30 02:49:06.103073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103017 5998 cpu_manager.go:414] "containerMap: removing container" podUID="820068d2-ba3f-4127-8f01-3b21c61acebd" containerName="istio-proxy" Oct 30 02:49:06.103073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103024 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820068d2-ba3f-4127-8f01-3b21c61acebd" containerName="istio-proxy" Oct 30 02:49:06.103073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103037 5998 cpu_manager.go:414] "containerMap: removing container" podUID="10593def-555a-449b-9da3-597a153c257d" containerName="revoke-keys" Oct 30 02:49:06.103073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103042 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10593def-555a-449b-9da3-597a153c257d" containerName="revoke-keys" Oct 30 02:49:06.103073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103053 5998 cpu_manager.go:414] "containerMap: removing container" podUID="820068d2-ba3f-4127-8f01-3b21c61acebd" containerName="istio-proxy" Oct 30 02:49:06.103073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103060 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820068d2-ba3f-4127-8f01-3b21c61acebd" containerName="istio-proxy" Oct 30 02:49:06.103073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103074 5998 cpu_manager.go:414] "containerMap: removing container" podUID="57be1a46-2149-46f4-a048-b099d67d3a6b" containerName="istio-proxy" Oct 30 02:49:06.103073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103078 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57be1a46-2149-46f4-a048-b099d67d3a6b" containerName="istio-proxy" Oct 30 02:49:06.103073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103104 5998 cpu_manager.go:414] "containerMap: removing container" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103110 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103117 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ff075586-a6a8-4eed-819d-19c55abfa057" containerName="istio-proxy" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103122 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff075586-a6a8-4eed-819d-19c55abfa057" containerName="istio-proxy" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103133 5998 cpu_manager.go:414] "containerMap: removing container" podUID="5220aca0-19d7-444c-b92b-da5361f306dc" containerName="istio-proxy" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103152 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5220aca0-19d7-444c-b92b-da5361f306dc" containerName="istio-proxy" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103161 5998 cpu_manager.go:414] "containerMap: removing container" podUID="5220aca0-19d7-444c-b92b-da5361f306dc" containerName="istio-proxy" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103165 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5220aca0-19d7-444c-b92b-da5361f306dc" containerName="istio-proxy" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103172 5998 cpu_manager.go:414] "containerMap: removing container" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103177 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103188 5998 cpu_manager.go:414] "containerMap: removing container" podUID="d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" containerName="istio-proxy" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103192 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" containerName="istio-proxy" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103257 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="57be1a46-2149-46f4-a048-b099d67d3a6b" containerName="istio-proxy" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103265 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="10593def-555a-449b-9da3-597a153c257d" containerName="revoke-keys" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103274 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="820068d2-ba3f-4127-8f01-3b21c61acebd" containerName="istio-proxy" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103281 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="b13ac864-0406-4db5-b911-2851f86df2b6" containerName="istio-proxy" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103287 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="63663089-b7d3-4565-a95a-3b06e913271b" containerName="main" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103292 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff075586-a6a8-4eed-819d-19c55abfa057" containerName="istio-proxy" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103299 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="5220aca0-19d7-444c-b92b-da5361f306dc" containerName="istio-proxy" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103308 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="7076db80-f9a3-423f-b198-f0a7aba80bd7" containerName="istio-proxy" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103314 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="5220aca0-19d7-444c-b92b-da5361f306dc" containerName="istio-proxy" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103319 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc2afe3c-bb27-460f-a7a0-abf3934daf2d" containerName="main" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103325 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c8b61a-b6aa-4f36-86c5-bd5cb6532ff5" containerName="istio-proxy" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103332 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e218532-053c-41f2-8114-ab1385bbc759" containerName="main" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103337 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae44a76-84eb-475f-abea-de3828e1f033" containerName="main" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103342 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c758e0-df24-4751-9751-24d3050bdcbb" containerName="main" Oct 30 02:49:06.103706 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.103348 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="820068d2-ba3f-4127-8f01-3b21c61acebd" containerName="istio-proxy" Oct 30 02:49:06.106739 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.106721 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.118404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.118357 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4"] Oct 30 02:49:06.121875 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.121844 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.126506 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.126481 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5dmpm\"" Oct 30 02:49:06.139100 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.139056 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd"] Oct 30 02:49:06.158791 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.158757 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8da8b22-5586-419c-9378-03d3d2d2c47f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.158940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.158796 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ww97\" (UniqueName: \"kubernetes.io/projected/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-kube-api-access-5ww97\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.158940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.158816 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.158940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.158832 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da8b22-5586-419c-9378-03d3d2d2c47f-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.158940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.158928 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da8b22-5586-419c-9378-03d3d2d2c47f-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.159131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.158972 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8da8b22-5586-419c-9378-03d3d2d2c47f-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.159131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.158994 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8da8b22-5586-419c-9378-03d3d2d2c47f-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.159131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.159010 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8da8b22-5586-419c-9378-03d3d2d2c47f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.159131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.159035 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.159131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.159075 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.159473 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.159149 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r566q\" (UniqueName: \"kubernetes.io/projected/d8da8b22-5586-419c-9378-03d3d2d2c47f-kube-api-access-r566q\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.159473 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.159180 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.159473 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.159314 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8da8b22-5586-419c-9378-03d3d2d2c47f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.159473 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.159400 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.159473 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.159438 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.159793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.159489 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8da8b22-5586-419c-9378-03d3d2d2c47f-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.159793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.159511 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.159793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.159536 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.172146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.172057 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4"] Oct 30 02:49:06.260438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.260398 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.260657 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.260449 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.260657 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.260487 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8da8b22-5586-419c-9378-03d3d2d2c47f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.260657 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.260609 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ww97\" (UniqueName: \"kubernetes.io/projected/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-kube-api-access-5ww97\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.260891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.260658 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.260891 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.260825 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da8b22-5586-419c-9378-03d3d2d2c47f-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.261029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.260921 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da8b22-5586-419c-9378-03d3d2d2c47f-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.261029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.260931 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.261029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.260970 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8da8b22-5586-419c-9378-03d3d2d2c47f-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.261029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.260974 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.261029 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.261006 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8da8b22-5586-419c-9378-03d3d2d2c47f-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.261336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.261031 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8da8b22-5586-419c-9378-03d3d2d2c47f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.261336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.261068 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.261336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.261123 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.261336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.261165 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r566q\" (UniqueName: \"kubernetes.io/projected/d8da8b22-5586-419c-9378-03d3d2d2c47f-kube-api-access-r566q\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.261336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.261187 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da8b22-5586-419c-9378-03d3d2d2c47f-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.261336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.261197 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.261336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.261234 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8da8b22-5586-419c-9378-03d3d2d2c47f-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.261336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.261247 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8da8b22-5586-419c-9378-03d3d2d2c47f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.261336 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.261274 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8da8b22-5586-419c-9378-03d3d2d2c47f-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.261890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.261513 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.261890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.261543 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.261890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.261604 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.261890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.261661 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.261890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.261709 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8da8b22-5586-419c-9378-03d3d2d2c47f-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.262222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.261935 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8da8b22-5586-419c-9378-03d3d2d2c47f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.262222 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.262047 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8da8b22-5586-419c-9378-03d3d2d2c47f-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.262436 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.262411 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.263344 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.263317 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8da8b22-5586-419c-9378-03d3d2d2c47f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.264194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.264159 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8da8b22-5586-419c-9378-03d3d2d2c47f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.264194 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.264184 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.264380 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.264357 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.281826 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.281796 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.282072 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.282040 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ww97\" (UniqueName: \"kubernetes.io/projected/0e843d95-ed87-4ea3-9ce8-d3f0103d9a80-kube-api-access-5ww97\") pod \"maas-default-gateway-openshift-default-75d7c785f-b95w4\" (UID: \"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.282223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.282158 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8da8b22-5586-419c-9378-03d3d2d2c47f-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.282280 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.282237 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r566q\" (UniqueName: \"kubernetes.io/projected/d8da8b22-5586-419c-9378-03d3d2d2c47f-kube-api-access-r566q\") pod \"maas-default-gateway-openshift-default-75d7c785f-28ssd\" (UID: \"d8da8b22-5586-419c-9378-03d3d2d2c47f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.420025 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.419987 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:06.434316 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.434128 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:06.762730 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.762626 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd"] Oct 30 02:49:06.765442 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.765406 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4"] Oct 30 02:49:06.767763 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:49:06.767732 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8da8b22_5586_419c_9378_03d3d2d2c47f.slice/crio-e2783b5f1252e9413ade39ce33a6856ba5cd98b5b3c43e3efc3faf795629d926 WatchSource:0}: Error finding container e2783b5f1252e9413ade39ce33a6856ba5cd98b5b3c43e3efc3faf795629d926: Status 404 returned error can't find the container with id e2783b5f1252e9413ade39ce33a6856ba5cd98b5b3c43e3efc3faf795629d926 Oct 30 02:49:06.770000 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.769968 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:49:06.770124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.770047 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:49:06.770124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.770117 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:49:06.781366 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:49:06.781339 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e843d95_ed87_4ea3_9ce8_d3f0103d9a80.slice/crio-9ee4d4968e5b8b804a6a5dc6dee8fe2eaf59f4f9b87f4be4c98bb89f211daab6 WatchSource:0}: Error finding container 9ee4d4968e5b8b804a6a5dc6dee8fe2eaf59f4f9b87f4be4c98bb89f211daab6: Status 404 returned error can't find the container with id 9ee4d4968e5b8b804a6a5dc6dee8fe2eaf59f4f9b87f4be4c98bb89f211daab6 Oct 30 02:49:06.783554 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.783524 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:49:06.783641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.783586 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:49:06.783641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:06.783616 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:49:07.100169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:07.099183 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" event={"ID":"d8da8b22-5586-419c-9378-03d3d2d2c47f","Type":"ContainerStarted","Data":"ccd9ab2983a072beb68ab8ed9e208a654fe5d48915059770a2b7f9109370a075"} Oct 30 02:49:07.100169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:07.099292 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" event={"ID":"d8da8b22-5586-419c-9378-03d3d2d2c47f","Type":"ContainerStarted","Data":"e2783b5f1252e9413ade39ce33a6856ba5cd98b5b3c43e3efc3faf795629d926"} Oct 30 02:49:07.103122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:07.101282 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" event={"ID":"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80","Type":"ContainerStarted","Data":"df88897694c878edeb9011cf5d6693411173479fd51e403e8373b5ba04fe4bb7"} Oct 30 02:49:07.103122 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:07.101338 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" event={"ID":"0e843d95-ed87-4ea3-9ce8-d3f0103d9a80","Type":"ContainerStarted","Data":"9ee4d4968e5b8b804a6a5dc6dee8fe2eaf59f4f9b87f4be4c98bb89f211daab6"} Oct 30 02:49:07.143313 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:07.143247 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" podStartSLOduration=1.1432277690000001 podStartE2EDuration="1.143227769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:49:07.140902154 +0000 UTC m=+4776.719693236" watchObservedRunningTime="2026-10-30 02:49:07.143227769 +0000 UTC m=+4776.722018851" Oct 30 02:49:07.187516 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:07.187452 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" podStartSLOduration=1.18743213 podStartE2EDuration="1.18743213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:49:07.182071023 +0000 UTC m=+4776.760862104" watchObservedRunningTime="2026-10-30 02:49:07.18743213 +0000 UTC m=+4776.766223212" Oct 30 02:49:07.422113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:07.422020 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-28ssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Oct 30 02:49:07.422382 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:07.422117 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" podUID="d8da8b22-5586-419c-9378-03d3d2d2c47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Oct 30 02:49:07.422503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:07.422431 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:07.441113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:07.440167 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:07.441113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:07.440357 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-b95w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body= Oct 30 02:49:07.441113 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:07.440410 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" podUID="0e843d95-ed87-4ea3-9ce8-d3f0103d9a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" Oct 30 02:49:08.422017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:08.421552 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-28ssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Oct 30 02:49:08.422017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:08.421617 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" podUID="d8da8b22-5586-419c-9378-03d3d2d2c47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Oct 30 02:49:08.436662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:08.436526 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-b95w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body= Oct 30 02:49:08.436662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:08.436595 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" podUID="0e843d95-ed87-4ea3-9ce8-d3f0103d9a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" Oct 30 02:49:09.424114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:09.422559 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-28ssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Oct 30 02:49:09.424114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:09.422627 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" podUID="d8da8b22-5586-419c-9378-03d3d2d2c47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Oct 30 02:49:09.438114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:09.437519 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-b95w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body= Oct 30 02:49:09.438114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:09.437600 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" podUID="0e843d95-ed87-4ea3-9ce8-d3f0103d9a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" Oct 30 02:49:10.421114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:10.420527 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-28ssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Oct 30 02:49:10.421114 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:10.420620 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" podUID="d8da8b22-5586-419c-9378-03d3d2d2c47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Oct 30 02:49:10.435561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:10.435519 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-b95w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body= Oct 30 02:49:10.437316 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:10.437278 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" podUID="0e843d95-ed87-4ea3-9ce8-d3f0103d9a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" Oct 30 02:49:11.420548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:11.420503 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-28ssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Oct 30 02:49:11.420894 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:11.420857 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" podUID="d8da8b22-5586-419c-9378-03d3d2d2c47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Oct 30 02:49:11.435554 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:11.435513 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-b95w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body= Oct 30 02:49:11.435826 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:11.435798 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" podUID="0e843d95-ed87-4ea3-9ce8-d3f0103d9a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" Oct 30 02:49:12.423555 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:12.423511 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-28ssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Oct 30 02:49:12.423915 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:12.423878 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" podUID="d8da8b22-5586-419c-9378-03d3d2d2c47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Oct 30 02:49:12.436446 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:12.436406 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-b95w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body= Oct 30 02:49:12.437021 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:12.436986 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" podUID="0e843d95-ed87-4ea3-9ce8-d3f0103d9a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" Oct 30 02:49:13.420420 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:13.420382 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-28ssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Oct 30 02:49:13.420631 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:13.420442 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" podUID="d8da8b22-5586-419c-9378-03d3d2d2c47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Oct 30 02:49:13.435257 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:13.435220 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-b95w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body= Oct 30 02:49:13.435404 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:13.435279 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" podUID="0e843d95-ed87-4ea3-9ce8-d3f0103d9a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" Oct 30 02:49:14.421169 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:14.421117 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-28ssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Oct 30 02:49:14.421600 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:14.421198 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" podUID="d8da8b22-5586-419c-9378-03d3d2d2c47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Oct 30 02:49:14.435498 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:14.435455 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-b95w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body= Oct 30 02:49:14.435662 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:14.435522 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" podUID="0e843d95-ed87-4ea3-9ce8-d3f0103d9a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" Oct 30 02:49:14.894362 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:14.894245 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:49:15.425182 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:15.425146 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:15.425703 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:15.425425 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:15.426340 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:15.426314 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" Oct 30 02:49:15.438669 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:15.438645 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:15.438935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:15.438915 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:15.439552 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:15.439534 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" Oct 30 02:49:17.333801 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:17.333766 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:49:21.144050 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.144013 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr"] Oct 30 02:49:21.147868 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.147838 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.178218 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.178181 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr"] Oct 30 02:49:21.257516 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.257482 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk85x\" (UniqueName: \"kubernetes.io/projected/961683fb-469d-4d4a-9463-269eb264251a-kube-api-access-wk85x\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.257735 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.257565 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961683fb-469d-4d4a-9463-269eb264251a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.257735 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.257641 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961683fb-469d-4d4a-9463-269eb264251a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.257735 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.257712 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961683fb-469d-4d4a-9463-269eb264251a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.257932 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.257761 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961683fb-469d-4d4a-9463-269eb264251a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.257932 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.257791 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961683fb-469d-4d4a-9463-269eb264251a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.257932 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.257889 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961683fb-469d-4d4a-9463-269eb264251a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.257932 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.257923 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961683fb-469d-4d4a-9463-269eb264251a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.258147 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.257949 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961683fb-469d-4d4a-9463-269eb264251a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.359423 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.359383 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961683fb-469d-4d4a-9463-269eb264251a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.359634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.359432 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk85x\" (UniqueName: \"kubernetes.io/projected/961683fb-469d-4d4a-9463-269eb264251a-kube-api-access-wk85x\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.359634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.359500 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961683fb-469d-4d4a-9463-269eb264251a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.359634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.359530 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961683fb-469d-4d4a-9463-269eb264251a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.359634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.359559 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961683fb-469d-4d4a-9463-269eb264251a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.359634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.359603 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961683fb-469d-4d4a-9463-269eb264251a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.359935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.359727 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961683fb-469d-4d4a-9463-269eb264251a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.359935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.359804 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961683fb-469d-4d4a-9463-269eb264251a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.359935 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.359895 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961683fb-469d-4d4a-9463-269eb264251a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.360151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.360055 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961683fb-469d-4d4a-9463-269eb264251a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.360151 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.360121 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961683fb-469d-4d4a-9463-269eb264251a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.360257 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.360146 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961683fb-469d-4d4a-9463-269eb264251a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.360333 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.360312 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961683fb-469d-4d4a-9463-269eb264251a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.360406 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.360387 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961683fb-469d-4d4a-9463-269eb264251a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.361947 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.361921 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961683fb-469d-4d4a-9463-269eb264251a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.362101 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.362072 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961683fb-469d-4d4a-9463-269eb264251a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.381163 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.381129 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961683fb-469d-4d4a-9463-269eb264251a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.381450 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.381425 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk85x\" (UniqueName: \"kubernetes.io/projected/961683fb-469d-4d4a-9463-269eb264251a-kube-api-access-wk85x\") pod \"maas-default-gateway-openshift-default-75d7c785f-ppvtr\" (UID: \"961683fb-469d-4d4a-9463-269eb264251a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.461461 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.461426 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:21.778017 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.777922 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr"] Oct 30 02:49:21.781972 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:49:21.781940 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961683fb_469d_4d4a_9463_269eb264251a.slice/crio-3593cc8c518592fda6bb6e6af70c78149e89fd5dcc3da130646e9d8da55a924b WatchSource:0}: Error finding container 3593cc8c518592fda6bb6e6af70c78149e89fd5dcc3da130646e9d8da55a924b: Status 404 returned error can't find the container with id 3593cc8c518592fda6bb6e6af70c78149e89fd5dcc3da130646e9d8da55a924b Oct 30 02:49:21.784175 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.784138 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:49:21.784236 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.784206 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:49:21.784284 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:21.784234 5998 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 02:49:22.233076 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:22.233031 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" event={"ID":"961683fb-469d-4d4a-9463-269eb264251a","Type":"ContainerStarted","Data":"eccd953baf959a65aee40e50143caff98f7b26e261f811a63bd79285d56492f3"} Oct 30 02:49:22.233610 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:22.233228 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" event={"ID":"961683fb-469d-4d4a-9463-269eb264251a","Type":"ContainerStarted","Data":"3593cc8c518592fda6bb6e6af70c78149e89fd5dcc3da130646e9d8da55a924b"} Oct 30 02:49:22.277579 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:22.277530 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" podStartSLOduration=1.277515417 podStartE2EDuration="1.277515417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:49:22.275961616 +0000 UTC m=+4791.854752699" watchObservedRunningTime="2026-10-30 02:49:22.277515417 +0000 UTC m=+4791.856306497" Oct 30 02:49:22.462481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:22.462438 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:22.463987 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:22.463955 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ppvtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" start-of-body= Oct 30 02:49:22.464153 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:22.464006 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" podUID="961683fb-469d-4d4a-9463-269eb264251a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" Oct 30 02:49:23.462537 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:23.462500 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ppvtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" start-of-body= Oct 30 02:49:23.462973 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:23.462552 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" podUID="961683fb-469d-4d4a-9463-269eb264251a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" Oct 30 02:49:24.462548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:24.462511 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ppvtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" start-of-body= Oct 30 02:49:24.462992 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:24.462580 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" podUID="961683fb-469d-4d4a-9463-269eb264251a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" Oct 30 02:49:25.239437 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:49:25.239400 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:49:25.462768 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:25.462724 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ppvtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" start-of-body= Oct 30 02:49:25.463240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:25.462785 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" podUID="961683fb-469d-4d4a-9463-269eb264251a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" Oct 30 02:49:26.462658 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:26.462621 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ppvtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" start-of-body= Oct 30 02:49:26.462869 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:26.462679 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" podUID="961683fb-469d-4d4a-9463-269eb264251a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" Oct 30 02:49:27.131689 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:27.131649 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:49:27.237247 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:27.237208 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:49:27.461865 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:27.461826 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ppvtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" start-of-body= Oct 30 02:49:27.462061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:27.461889 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" podUID="961683fb-469d-4d4a-9463-269eb264251a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" Oct 30 02:49:28.461954 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:28.461917 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ppvtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" start-of-body= Oct 30 02:49:28.462408 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:28.461971 5998 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" podUID="961683fb-469d-4d4a-9463-269eb264251a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.77:15021/healthz/ready\": dial tcp 10.133.1.77:15021: connect: connection refused" Oct 30 02:49:29.952663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:29.952625 5998 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:29.953213 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:29.953125 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:29.954167 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:29.954138 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" Oct 30 02:49:35.229357 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:35.229319 5998 scope.go:122] "RemoveContainer" containerID="eb08334a42960d3c89fa9644c522ca309957bade642974f535f21b5d73608fb2" Oct 30 02:49:35.238542 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:35.238517 5998 scope.go:122] "RemoveContainer" containerID="fa26205384f46214855ec05ea7b115d532a46a1d70d97add399f28f44613f6c3" Oct 30 02:49:35.246939 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:35.246918 5998 scope.go:122] "RemoveContainer" containerID="db3d735fe337c4e9e99799dc258699178ac77443991b489d26ab2df7d51b745c" Oct 30 02:49:35.255149 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:35.255124 5998 scope.go:122] "RemoveContainer" containerID="e62fcdb35e6ad35b0e3ac5e5eae4cb38d6b47bd08f557ef84e2160690c48f344" Oct 30 02:49:38.403635 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:38.403599 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:49:41.189667 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:41.189628 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:49:42.009890 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:42.009855 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:49:50.760422 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:50.760379 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:49:50.869714 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:50.869672 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:49:52.908160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:52.908124 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:49:54.095121 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:54.095049 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:49:55.224660 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:49:55.224590 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:49:57.042046 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:57.042002 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp"] Oct 30 02:49:57.068053 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:57.068006 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99cb651-4gtdp"] Oct 30 02:49:58.028983 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:58.028927 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x"] Oct 30 02:49:58.039550 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:58.039514 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3625c7d7-fpd8x"] Oct 30 02:49:58.575132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:58.573723 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f42c813-e505-4819-b8b5-f080cf92ffd6" path="/var/lib/kubelet/pods/4f42c813-e505-4819-b8b5-f080cf92ffd6/volumes" Oct 30 02:49:58.575132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:49:58.574424 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3508546-4215-44b4-9f02-4c90461c1083" path="/var/lib/kubelet/pods/c3508546-4215-44b4-9f02-4c90461c1083/volumes" Oct 30 02:50:00.035809 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:00.035769 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m"] Oct 30 02:50:00.049539 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:00.049454 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c437787-dlr6m"] Oct 30 02:50:00.538421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:00.538384 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85df47f5-8e2a-4eee-b6d8-3ec2636def64" path="/var/lib/kubelet/pods/85df47f5-8e2a-4eee-b6d8-3ec2636def64/volumes" Oct 30 02:50:00.698240 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:00.698040 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:50:02.049712 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:02.049673 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:50:02.172648 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:02.172615 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:50:06.162801 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:06.162754 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:50:09.034468 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:09.034421 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc"] Oct 30 02:50:09.044580 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:09.044538 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bb3ae5d-drprc"] Oct 30 02:50:10.497125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:10.497067 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:50:10.537530 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:10.537489 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1d4b059-e455-4127-bccc-da21bf6e0680" path="/var/lib/kubelet/pods/e1d4b059-e455-4127-bccc-da21bf6e0680/volumes" Oct 30 02:50:23.033117 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:23.032384 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh"] Oct 30 02:50:23.044032 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:23.043897 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdbdb098-jlbwh"] Oct 30 02:50:24.537027 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:24.536988 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ca672a-fa36-4182-8cac-59d4d8b992b6" path="/var/lib/kubelet/pods/c5ca672a-fa36-4182-8cac-59d4d8b992b6/volumes" Oct 30 02:50:25.216756 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:50:25.216715 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:50:33.031621 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:33.031581 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv"] Oct 30 02:50:33.042126 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:33.042077 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa868ea5-mkpcv"] Oct 30 02:50:34.537572 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:34.537530 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dffe7505-ba98-42c5-aba9-0ef9539b16e4" path="/var/lib/kubelet/pods/dffe7505-ba98-42c5-aba9-0ef9539b16e4/volumes" Oct 30 02:50:42.162124 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:42.162031 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:50:46.179290 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:46.179248 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:50:49.656522 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:49.656478 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:50:50.023201 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:50.023168 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:50:51.802007 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:51.801965 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:50:52.272270 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:50:52.272230 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:50:55.392411 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:50:55.392349 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:51:21.740955 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:51:21.740911 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:51:21.848713 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:51:21.848680 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:51:25.215759 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:51:25.215717 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:51:30.285486 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:51:30.285442 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:51:35.298373 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:51:35.298336 5998 scope.go:122] "RemoveContainer" containerID="bc99b9663c5d261f5fda83410fd5d8057491f5ab95ccaf447216eb139ecb1ef6" Oct 30 02:51:35.307388 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:51:35.307363 5998 scope.go:122] "RemoveContainer" containerID="41dcbf62abdd821af8a2af72b999cca8dcaa616550816de977a1b5725e0a194d" Oct 30 02:51:35.315821 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:51:35.315796 5998 scope.go:122] "RemoveContainer" containerID="97a89d83f97b9e24acfb4330a8a3669db7c78d8c11e40ec65b7be4ad51801cc5" Oct 30 02:51:35.323661 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:51:35.323634 5998 scope.go:122] "RemoveContainer" containerID="a86e3538c48c1b8c7d1639f70a6902ede0fb06d530566567fddfc00823a09e1b" Oct 30 02:51:35.334710 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:51:35.334686 5998 scope.go:122] "RemoveContainer" containerID="78a9a6e58ed31ddf6775eb00e4d08ae9e3effe18e7c7563d8420e0db3202835e" Oct 30 02:51:35.342969 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:51:35.342945 5998 scope.go:122] "RemoveContainer" containerID="d4936c4fb71ec7ce1b5fc815949b9fb7597bbc180a6752c3757a85e20a3e544e" Oct 30 02:51:42.286744 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:51:42.286707 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:51:49.236725 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:51:49.236682 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:51:49.620516 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:51:49.620419 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:51:49.829969 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:51:49.829928 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:51:55.288186 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:51:55.288022 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:51:56.616317 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:51:56.616276 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:52:00.044256 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:00.044212 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn"] Oct 30 02:52:00.054146 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:00.054106 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16862d1f-4qhtn"] Oct 30 02:52:00.539078 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:00.539039 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10593def-555a-449b-9da3-597a153c257d" path="/var/lib/kubelet/pods/10593def-555a-449b-9da3-597a153c257d/volumes" Oct 30 02:52:04.376773 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:04.376740 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:05.012793 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:05.012734 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:05.159666 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:05.159625 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:08.056283 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:08.056243 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:08.392033 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:08.391936 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:09.564561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:09.564524 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:09.769077 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:09.769047 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:09.936380 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:09.936341 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:11.488011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:11.487974 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:13.090751 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:13.090713 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:13.500190 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:13.500138 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:52:14.683160 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:14.683120 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:16.283061 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:16.283027 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:17.893131 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:17.893074 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:19.593692 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:19.593635 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:21.096623 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:21.096584 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:22.831784 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:22.831711 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:22.975148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:22.975115 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:23.629548 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:23.629483 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:23.762686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:23.762643 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:23.886909 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:23.886796 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:24.521815 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:24.521773 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:25.217038 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:52:25.216994 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:52:26.080197 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:26.080139 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:27.207365 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:27.207321 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:27.685561 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:27.685517 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:29.271464 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:29.271423 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:29.806873 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:29.806833 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:52:30.984366 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:30.984323 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:32.489018 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:32.488974 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:33.971969 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:33.971922 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:35.583358 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:35.583322 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:37.073977 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:37.073935 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:38.682047 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:38.682004 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:40.208071 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:40.208035 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:42.826772 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:42.826719 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:42.942148 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:42.942099 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:43.606185 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:43.606147 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:44.875782 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:44.875733 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:52:45.338840 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:45.338802 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:47.126302 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:47.126222 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:48.682677 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:48.682641 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:50.273101 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:50.273048 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:51.788339 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:51.788303 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:53.286601 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:53.286558 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:54.792718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:54.792682 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:55.216794 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:52:55.216755 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:52:56.368202 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:56.368166 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:57.785203 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:57.785162 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd"] Oct 30 02:52:57.788676 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:57.788648 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd" Oct 30 02:52:57.825070 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:57.825032 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:52:57.825070 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:57.825054 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 02:52:57.825070 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:57.825059 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 02:52:57.836857 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:57.836829 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227da02f-35df-4ab7-89a4-cd355a02ff74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b10cf7d-6mwrd\" (UID: \"227da02f-35df-4ab7-89a4-cd355a02ff74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd" Oct 30 02:52:57.837042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:57.836896 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227da02f-35df-4ab7-89a4-cd355a02ff74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b10cf7d-6mwrd\" (UID: \"227da02f-35df-4ab7-89a4-cd355a02ff74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd" Oct 30 02:52:57.837042 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:57.836971 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndpmd\" (UniqueName: \"kubernetes.io/projected/227da02f-35df-4ab7-89a4-cd355a02ff74-kube-api-access-ndpmd\") pod \"maas-api-revoke-sub-1b10cf7d-6mwrd\" (UID: \"227da02f-35df-4ab7-89a4-cd355a02ff74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd" Oct 30 02:52:57.839636 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:57.839618 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 02:52:57.856011 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:57.855973 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd"] Oct 30 02:52:57.937481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:57.937442 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227da02f-35df-4ab7-89a4-cd355a02ff74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b10cf7d-6mwrd\" (UID: \"227da02f-35df-4ab7-89a4-cd355a02ff74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd" Oct 30 02:52:57.937680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:57.937497 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndpmd\" (UniqueName: \"kubernetes.io/projected/227da02f-35df-4ab7-89a4-cd355a02ff74-kube-api-access-ndpmd\") pod \"maas-api-revoke-sub-1b10cf7d-6mwrd\" (UID: \"227da02f-35df-4ab7-89a4-cd355a02ff74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd" Oct 30 02:52:57.937680 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:57.937550 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227da02f-35df-4ab7-89a4-cd355a02ff74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b10cf7d-6mwrd\" (UID: \"227da02f-35df-4ab7-89a4-cd355a02ff74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd" Oct 30 02:52:57.938230 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:57.938206 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227da02f-35df-4ab7-89a4-cd355a02ff74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b10cf7d-6mwrd\" (UID: \"227da02f-35df-4ab7-89a4-cd355a02ff74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd" Oct 30 02:52:57.939824 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:57.939800 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227da02f-35df-4ab7-89a4-cd355a02ff74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b10cf7d-6mwrd\" (UID: \"227da02f-35df-4ab7-89a4-cd355a02ff74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd" Oct 30 02:52:57.959503 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:57.959467 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndpmd\" (UniqueName: \"kubernetes.io/projected/227da02f-35df-4ab7-89a4-cd355a02ff74-kube-api-access-ndpmd\") pod \"maas-api-revoke-sub-1b10cf7d-6mwrd\" (UID: \"227da02f-35df-4ab7-89a4-cd355a02ff74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd" Oct 30 02:52:58.101940 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:58.101847 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd" Oct 30 02:52:58.115041 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:58.115007 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:52:58.411998 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:58.411955 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd"] Oct 30 02:52:58.416975 ip-10-0-130-0 kubenswrapper[5998]: W1030 02:52:58.416949 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227da02f_35df_4ab7_89a4_cd355a02ff74.slice/crio-0f08714154fb2ae21b54ee58f32a19357c3f573e62d790e6078f3eaa087a4a9b WatchSource:0}: Error finding container 0f08714154fb2ae21b54ee58f32a19357c3f573e62d790e6078f3eaa087a4a9b: Status 404 returned error can't find the container with id 0f08714154fb2ae21b54ee58f32a19357c3f573e62d790e6078f3eaa087a4a9b Oct 30 02:52:58.659144 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:58.659081 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd" event={"ID":"227da02f-35df-4ab7-89a4-cd355a02ff74","Type":"ContainerStarted","Data":"0f08714154fb2ae21b54ee58f32a19357c3f573e62d790e6078f3eaa087a4a9b"} Oct 30 02:52:59.665736 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:59.665700 5998 generic.go:356] "Generic (PLEG): container finished" podID="227da02f-35df-4ab7-89a4-cd355a02ff74" containerID="079efa65ee10efa7ec5da9ddebfdc11d05a7761228cbac8962990a870ddd820c" exitCode=0 Oct 30 02:52:59.666171 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:59.665784 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd" event={"ID":"227da02f-35df-4ab7-89a4-cd355a02ff74","Type":"ContainerDied","Data":"079efa65ee10efa7ec5da9ddebfdc11d05a7761228cbac8962990a870ddd820c"} Oct 30 02:52:59.801640 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:52:59.801599 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:00.804520 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:00.804493 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd" Oct 30 02:53:00.868062 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:00.868023 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/227da02f-35df-4ab7-89a4-cd355a02ff74-kube-api-access-ndpmd\" (UniqueName: \"kubernetes.io/projected/227da02f-35df-4ab7-89a4-cd355a02ff74-kube-api-access-ndpmd\") pod \"227da02f-35df-4ab7-89a4-cd355a02ff74\" (UID: \"227da02f-35df-4ab7-89a4-cd355a02ff74\") " Oct 30 02:53:00.868275 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:00.868077 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/227da02f-35df-4ab7-89a4-cd355a02ff74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227da02f-35df-4ab7-89a4-cd355a02ff74-maas-api-openshift-ca\") pod \"227da02f-35df-4ab7-89a4-cd355a02ff74\" (UID: \"227da02f-35df-4ab7-89a4-cd355a02ff74\") " Oct 30 02:53:00.868275 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:00.868215 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/227da02f-35df-4ab7-89a4-cd355a02ff74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227da02f-35df-4ab7-89a4-cd355a02ff74-maas-api-xks-ca\") pod \"227da02f-35df-4ab7-89a4-cd355a02ff74\" (UID: \"227da02f-35df-4ab7-89a4-cd355a02ff74\") " Oct 30 02:53:00.868491 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:00.868469 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227da02f-35df-4ab7-89a4-cd355a02ff74-maas-api-openshift-ca" pod "227da02f-35df-4ab7-89a4-cd355a02ff74" (UID: "227da02f-35df-4ab7-89a4-cd355a02ff74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:53:00.868663 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:00.868619 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227da02f-35df-4ab7-89a4-cd355a02ff74-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:53:00.870330 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:00.870299 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227da02f-35df-4ab7-89a4-cd355a02ff74-kube-api-access-ndpmd" pod "227da02f-35df-4ab7-89a4-cd355a02ff74" (UID: "227da02f-35df-4ab7-89a4-cd355a02ff74"). InnerVolumeSpecName "kube-api-access-ndpmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:53:00.870438 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:00.870311 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/227da02f-35df-4ab7-89a4-cd355a02ff74-maas-api-xks-ca" pod "227da02f-35df-4ab7-89a4-cd355a02ff74" (UID: "227da02f-35df-4ab7-89a4-cd355a02ff74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:53:00.970132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:00.970012 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227da02f-35df-4ab7-89a4-cd355a02ff74-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:53:00.970132 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:00.970048 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndpmd\" (UniqueName: \"kubernetes.io/projected/227da02f-35df-4ab7-89a4-cd355a02ff74-kube-api-access-ndpmd\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 02:53:01.407481 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:01.407395 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:01.679927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:01.679886 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd" event={"ID":"227da02f-35df-4ab7-89a4-cd355a02ff74","Type":"ContainerDied","Data":"0f08714154fb2ae21b54ee58f32a19357c3f573e62d790e6078f3eaa087a4a9b"} Oct 30 02:53:01.679927 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:01.679920 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd" Oct 30 02:53:01.680179 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:01.679943 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f08714154fb2ae21b54ee58f32a19357c3f573e62d790e6078f3eaa087a4a9b" Oct 30 02:53:02.980835 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:02.980796 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:04.489870 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:04.489817 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:05.989118 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:05.989026 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:07.491931 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:07.491885 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:08.555421 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:08.555385 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:09.037994 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:09.037940 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:09.175478 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:09.175447 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:09.196517 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:09.196471 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:53:10.767492 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:10.767443 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:12.275589 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:12.275538 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:13.589971 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:13.589935 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:13.775463 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:13.775424 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:14.637310 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:14.637276 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:14.745564 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:14.745525 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:15.453917 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:15.453872 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:15.723997 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:15.723868 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:16.834808 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:16.834773 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:19.270618 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:19.270574 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:53:21.884718 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:21.884675 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:21.996209 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:21.996173 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:22.558379 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:22.558339 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:22.958187 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:22.958148 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:25.217589 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:53:25.217545 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:53:27.181406 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:27.181370 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:29.565084 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:29.565044 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:53:35.390879 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:35.390840 5998 scope.go:122] "RemoveContainer" containerID="a7f9b22aba27325fcbabb1419c6bbace600e869eed200c1003751ccfcea6b3fa" Oct 30 02:53:42.560178 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:42.560076 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:44.145268 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:44.145230 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:46.467374 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:46.467334 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:53:50.535859 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:50.535818 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:51.640818 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:51.640778 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:53:55.215881 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:53:55.215839 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:53:58.003034 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:53:58.002990 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:54:07.473197 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:54:07.473153 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:54:13.091267 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:54:13.091230 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:54:13.196073 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:54:13.196039 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:54:17.328125 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:54:17.323589 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:54:19.482641 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:54:19.482603 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:54:23.423634 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:54:23.423589 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:54:25.217117 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:54:25.217063 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:54:26.365613 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:54:26.365572 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:54:38.296139 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:54:38.296084 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:54:42.725165 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:54:42.725116 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:54:44.752507 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:54:44.752462 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:54:55.216648 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:54:55.216601 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:54:57.194635 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:54:57.194590 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:55:19.860817 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:55:19.860729 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:55:25.215947 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:55:25.215899 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:55:41.974030 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:55:41.973993 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:55:42.145745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:55:42.145701 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:55:42.768405 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:55:42.768370 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:55:42.878701 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:55:42.878661 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:55:44.264223 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:55:44.264177 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:55:55.216059 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:55:55.216022 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:56:25.217167 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:56:25.217121 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:56:25.498430 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:56:25.498338 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:56:25.606934 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:56:25.606898 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:56:36.226687 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:56:36.226654 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:56:42.963760 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:56:42.963665 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:56:55.216120 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:56:55.216064 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:57:25.216271 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:57:25.216226 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:57:34.116729 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:57:34.116688 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:57:34.257332 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:57:34.257284 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:57:34.420863 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:57:34.420818 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:57:35.490795 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:57:35.490720 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:57:40.877469 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:57:40.877430 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:57:42.401301 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:57:42.401257 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:57:43.114242 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:57:43.114195 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:57:55.216045 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:57:55.215999 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:57:58.197256 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:57:58.197217 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:58:01.041745 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:58:01.041701 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd"] Oct 30 02:58:01.051686 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:58:01.051653 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b10cf7d-6mwrd"] Oct 30 02:58:02.539530 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:58:02.539490 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="227da02f-35df-4ab7-89a4-cd355a02ff74" path="/var/lib/kubelet/pods/227da02f-35df-4ab7-89a4-cd355a02ff74/volumes" Oct 30 02:58:02.766221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:58:02.766165 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:58:15.946779 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:58:15.946696 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:58:16.051068 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:58:16.051031 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:58:18.073867 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:58:18.073125 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:58:25.216118 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:58:25.216061 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:58:33.672544 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:58:33.672510 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:58:43.249659 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:58:43.249608 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:58:55.219959 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:58:55.219918 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:58:57.092689 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:58:57.092652 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:59:07.485221 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:59:07.485165 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:59:12.111820 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:59:12.111774 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:59:25.215787 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:59:25.215743 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:59:29.167980 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:59:29.167937 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:59:34.235393 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:59:34.235356 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:59:35.469454 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:59:35.469416 5998 scope.go:122] "RemoveContainer" containerID="079efa65ee10efa7ec5da9ddebfdc11d05a7761228cbac8962990a870ddd820c" Oct 30 02:59:38.467449 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:59:38.467402 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:59:43.389272 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:59:43.389155 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 02:59:50.696353 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:59:50.696288 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 02:59:55.218541 ip-10-0-130-0 kubenswrapper[5998]: E1030 02:59:55.218496 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 02:59:56.006933 ip-10-0-130-0 kubenswrapper[5998]: I1030 02:59:56.006835 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:00:04.296661 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:04.296619 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:00:05.876406 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:05.876361 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4"] Oct 30 03:00:05.877744 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:05.877634 5998 cpu_manager.go:414] "containerMap: removing container" podUID="227da02f-35df-4ab7-89a4-cd355a02ff74" containerName="revoke-keys" Oct 30 03:00:05.877925 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:05.877905 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="227da02f-35df-4ab7-89a4-cd355a02ff74" containerName="revoke-keys" Oct 30 03:00:05.878202 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:05.878185 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="227da02f-35df-4ab7-89a4-cd355a02ff74" containerName="revoke-keys" Oct 30 03:00:05.882186 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:05.882166 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4" Oct 30 03:00:05.947114 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:05.921418 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:00:05.947114 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:05.921478 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:00:05.947114 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:05.924670 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4"] Oct 30 03:00:05.947114 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:05.924854 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:00:05.947114 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:05.924975 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:00:05.997495 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:05.997265 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx8h5\" (UniqueName: \"kubernetes.io/projected/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-kube-api-access-cx8h5\") pod \"maas-api-revoke-sub-0b7e8edb-jrgm4\" (UID: \"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4" Oct 30 03:00:05.997495 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:05.997404 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-jrgm4\" (UID: \"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4" Oct 30 03:00:05.997495 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:05.997465 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-jrgm4\" (UID: \"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4" Oct 30 03:00:06.098452 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:06.098402 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-jrgm4\" (UID: \"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4" Oct 30 03:00:06.098660 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:06.098497 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-jrgm4\" (UID: \"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4" Oct 30 03:00:06.098660 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:06.098554 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx8h5\" (UniqueName: \"kubernetes.io/projected/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-kube-api-access-cx8h5\") pod \"maas-api-revoke-sub-0b7e8edb-jrgm4\" (UID: \"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4" Oct 30 03:00:06.099335 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:06.099304 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-jrgm4\" (UID: \"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4" Oct 30 03:00:06.128997 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:06.128913 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-jrgm4\" (UID: \"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4" Oct 30 03:00:06.137119 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:06.135119 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx8h5\" (UniqueName: \"kubernetes.io/projected/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-kube-api-access-cx8h5\") pod \"maas-api-revoke-sub-0b7e8edb-jrgm4\" (UID: \"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4" Oct 30 03:00:06.196388 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:06.196352 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4" Oct 30 03:00:06.699288 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:06.699227 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4"] Oct 30 03:00:07.299164 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:07.298630 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4" event={"ID":"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32","Type":"ContainerStarted","Data":"0ea42d9f36d61d9f6c79efb33ef0d3cf239c4e9ee7684d94e869d027b9744c30"} Oct 30 03:00:07.299164 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:07.298739 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4" event={"ID":"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32","Type":"ContainerStarted","Data":"7516990e9c68c38844a53e30f242863f6c8ea7715f5361381cf9b441a3e9d3db"} Oct 30 03:00:07.341665 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:07.338839 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4" podStartSLOduration=2.338820585 podStartE2EDuration="2.338820585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:00:07.338297282 +0000 UTC m=+5436.917088376" watchObservedRunningTime="2026-10-30 03:00:07.338820585 +0000 UTC m=+5436.917611666" Oct 30 03:00:08.306772 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:08.306681 5998 generic.go:356] "Generic (PLEG): container finished" podID="3dbcaf5a-6527-4624-a6bc-c161cc9a0a32" containerID="0ea42d9f36d61d9f6c79efb33ef0d3cf239c4e9ee7684d94e869d027b9744c30" exitCode=0 Oct 30 03:00:08.307184 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:08.306762 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4" event={"ID":"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32","Type":"ContainerDied","Data":"0ea42d9f36d61d9f6c79efb33ef0d3cf239c4e9ee7684d94e869d027b9744c30"} Oct 30 03:00:09.456952 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:09.456924 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4" Oct 30 03:00:09.586980 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:09.586885 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:00:09.602875 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:09.602841 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-kube-api-access-cx8h5\" (UniqueName: \"kubernetes.io/projected/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-kube-api-access-cx8h5\") pod \"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32\" (UID: \"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32\") " Oct 30 03:00:09.603050 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:09.602975 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-maas-api-xks-ca\") pod \"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32\" (UID: \"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32\") " Oct 30 03:00:09.603050 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:09.603040 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-maas-api-openshift-ca\") pod \"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32\" (UID: \"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32\") " Oct 30 03:00:09.603449 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:09.603423 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-maas-api-openshift-ca" pod "3dbcaf5a-6527-4624-a6bc-c161cc9a0a32" (UID: "3dbcaf5a-6527-4624-a6bc-c161cc9a0a32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:00:09.605140 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:09.605107 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-maas-api-xks-ca" pod "3dbcaf5a-6527-4624-a6bc-c161cc9a0a32" (UID: "3dbcaf5a-6527-4624-a6bc-c161cc9a0a32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:00:09.605272 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:09.605211 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-kube-api-access-cx8h5" pod "3dbcaf5a-6527-4624-a6bc-c161cc9a0a32" (UID: "3dbcaf5a-6527-4624-a6bc-c161cc9a0a32"). InnerVolumeSpecName "kube-api-access-cx8h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:00:09.704370 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:09.704325 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:00:09.704370 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:09.704363 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:00:09.704370 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:09.704372 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx8h5\" (UniqueName: \"kubernetes.io/projected/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32-kube-api-access-cx8h5\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:00:10.319517 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:10.319464 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4" event={"ID":"3dbcaf5a-6527-4624-a6bc-c161cc9a0a32","Type":"ContainerDied","Data":"7516990e9c68c38844a53e30f242863f6c8ea7715f5361381cf9b441a3e9d3db"} Oct 30 03:00:10.319517 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:10.319489 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4" Oct 30 03:00:10.319517 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:10.319520 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7516990e9c68c38844a53e30f242863f6c8ea7715f5361381cf9b441a3e9d3db" Oct 30 03:00:16.600836 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.600796 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf"] Oct 30 03:00:16.601350 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.601226 5998 cpu_manager.go:414] "containerMap: removing container" podUID="3dbcaf5a-6527-4624-a6bc-c161cc9a0a32" containerName="revoke-keys" Oct 30 03:00:16.601350 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.601241 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbcaf5a-6527-4624-a6bc-c161cc9a0a32" containerName="revoke-keys" Oct 30 03:00:16.601350 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.601322 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbcaf5a-6527-4624-a6bc-c161cc9a0a32" containerName="revoke-keys" Oct 30 03:00:16.605277 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.605251 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf" Oct 30 03:00:16.618167 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.618143 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:00:16.628653 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.628608 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:00:16.628653 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.628621 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:00:16.628653 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.628625 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:00:16.631278 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.631250 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf"] Oct 30 03:00:16.771136 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.771084 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8vwkf\" (UID: \"1359a1bd-f5ae-4843-bd73-6f9b2f289f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf" Oct 30 03:00:16.771338 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.771225 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzfcz\" (UniqueName: \"kubernetes.io/projected/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-kube-api-access-lzfcz\") pod \"maas-api-revoke-sub-3d4156f1-8vwkf\" (UID: \"1359a1bd-f5ae-4843-bd73-6f9b2f289f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf" Oct 30 03:00:16.771338 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.771283 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8vwkf\" (UID: \"1359a1bd-f5ae-4843-bd73-6f9b2f289f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf" Oct 30 03:00:16.872346 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.872255 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8vwkf\" (UID: \"1359a1bd-f5ae-4843-bd73-6f9b2f289f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf" Oct 30 03:00:16.872346 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.872321 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzfcz\" (UniqueName: \"kubernetes.io/projected/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-kube-api-access-lzfcz\") pod \"maas-api-revoke-sub-3d4156f1-8vwkf\" (UID: \"1359a1bd-f5ae-4843-bd73-6f9b2f289f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf" Oct 30 03:00:16.872557 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.872352 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8vwkf\" (UID: \"1359a1bd-f5ae-4843-bd73-6f9b2f289f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf" Oct 30 03:00:16.873027 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.873007 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8vwkf\" (UID: \"1359a1bd-f5ae-4843-bd73-6f9b2f289f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf" Oct 30 03:00:16.874708 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.874687 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8vwkf\" (UID: \"1359a1bd-f5ae-4843-bd73-6f9b2f289f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf" Oct 30 03:00:16.895552 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.895518 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzfcz\" (UniqueName: \"kubernetes.io/projected/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-kube-api-access-lzfcz\") pod \"maas-api-revoke-sub-3d4156f1-8vwkf\" (UID: \"1359a1bd-f5ae-4843-bd73-6f9b2f289f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf" Oct 30 03:00:16.918469 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:16.918438 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf" Oct 30 03:00:17.220372 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:17.219873 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf"] Oct 30 03:00:17.224651 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:00:17.224609 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1359a1bd_f5ae_4843_bd73_6f9b2f289f07.slice/crio-2665ce6aba3d65e3db76569bc2bd1832ca27a3b580a1a497ebe7cf8fbf31e7c3 WatchSource:0}: Error finding container 2665ce6aba3d65e3db76569bc2bd1832ca27a3b580a1a497ebe7cf8fbf31e7c3: Status 404 returned error can't find the container with id 2665ce6aba3d65e3db76569bc2bd1832ca27a3b580a1a497ebe7cf8fbf31e7c3 Oct 30 03:00:17.356579 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:17.356540 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf" event={"ID":"1359a1bd-f5ae-4843-bd73-6f9b2f289f07","Type":"ContainerStarted","Data":"2665ce6aba3d65e3db76569bc2bd1832ca27a3b580a1a497ebe7cf8fbf31e7c3"} Oct 30 03:00:18.363398 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:18.363306 5998 generic.go:356] "Generic (PLEG): container finished" podID="1359a1bd-f5ae-4843-bd73-6f9b2f289f07" containerID="9533f30dc5dd5ea12d16c699085c56beda3afa8688d093feec5d907706e62794" exitCode=0 Oct 30 03:00:18.363398 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:18.363347 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf" event={"ID":"1359a1bd-f5ae-4843-bd73-6f9b2f289f07","Type":"ContainerDied","Data":"9533f30dc5dd5ea12d16c699085c56beda3afa8688d093feec5d907706e62794"} Oct 30 03:00:19.504948 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:19.504915 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf" Oct 30 03:00:19.700479 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:19.700438 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-maas-api-openshift-ca\") pod \"1359a1bd-f5ae-4843-bd73-6f9b2f289f07\" (UID: \"1359a1bd-f5ae-4843-bd73-6f9b2f289f07\") " Oct 30 03:00:19.700678 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:19.700622 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-maas-api-xks-ca\") pod \"1359a1bd-f5ae-4843-bd73-6f9b2f289f07\" (UID: \"1359a1bd-f5ae-4843-bd73-6f9b2f289f07\") " Oct 30 03:00:19.700778 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:19.700703 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-kube-api-access-lzfcz\" (UniqueName: \"kubernetes.io/projected/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-kube-api-access-lzfcz\") pod \"1359a1bd-f5ae-4843-bd73-6f9b2f289f07\" (UID: \"1359a1bd-f5ae-4843-bd73-6f9b2f289f07\") " Oct 30 03:00:19.700841 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:19.700819 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-maas-api-openshift-ca" pod "1359a1bd-f5ae-4843-bd73-6f9b2f289f07" (UID: "1359a1bd-f5ae-4843-bd73-6f9b2f289f07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:00:19.701029 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:19.701006 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:00:19.703439 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:19.703408 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-kube-api-access-lzfcz" pod "1359a1bd-f5ae-4843-bd73-6f9b2f289f07" (UID: "1359a1bd-f5ae-4843-bd73-6f9b2f289f07"). InnerVolumeSpecName "kube-api-access-lzfcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:00:19.703533 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:19.703498 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-maas-api-xks-ca" pod "1359a1bd-f5ae-4843-bd73-6f9b2f289f07" (UID: "1359a1bd-f5ae-4843-bd73-6f9b2f289f07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:00:19.801972 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:19.801922 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzfcz\" (UniqueName: \"kubernetes.io/projected/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-kube-api-access-lzfcz\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:00:19.801972 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:19.801957 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1359a1bd-f5ae-4843-bd73-6f9b2f289f07-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:00:20.374843 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:20.374808 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf" Oct 30 03:00:20.375041 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:20.374849 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf" event={"ID":"1359a1bd-f5ae-4843-bd73-6f9b2f289f07","Type":"ContainerDied","Data":"2665ce6aba3d65e3db76569bc2bd1832ca27a3b580a1a497ebe7cf8fbf31e7c3"} Oct 30 03:00:20.375041 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:20.374902 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2665ce6aba3d65e3db76569bc2bd1832ca27a3b580a1a497ebe7cf8fbf31e7c3" Oct 30 03:00:25.215933 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:00:25.215889 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:00:28.785923 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:28.785850 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:00:31.468316 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:31.468273 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-28ssd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:00:31.468952 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:31.468914 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" podUID="d8da8b22-5586-419c-9378-03d3d2d2c47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:00:31.484320 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:31.484206 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-b95w4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:00:31.484320 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:31.484257 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" podUID="0e843d95-ed87-4ea3-9ce8-d3f0103d9a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:00:35.185030 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:35.184987 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:00:35.890997 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:35.890958 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf"] Oct 30 03:00:35.901272 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:35.901242 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8vwkf"] Oct 30 03:00:36.537485 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:36.537448 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1359a1bd-f5ae-4843-bd73-6f9b2f289f07" path="/var/lib/kubelet/pods/1359a1bd-f5ae-4843-bd73-6f9b2f289f07/volumes" Oct 30 03:00:43.478136 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:43.478061 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:00:47.991669 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:47.991630 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:00:53.188635 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:53.188597 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:00:55.220264 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:00:55.220220 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:00:58.882628 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:00:58.882587 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:01:01.473793 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:01.473265 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-28ssd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:01:01.473793 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:01.473348 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" podUID="d8da8b22-5586-419c-9378-03d3d2d2c47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:01:01.487279 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:01.487242 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-b95w4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:01:01.487483 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:01.487422 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" podUID="0e843d95-ed87-4ea3-9ce8-d3f0103d9a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:01:16.880540 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:16.880444 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:01:25.216555 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:01:25.216511 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:01:27.199683 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:27.199630 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:01:43.261743 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:43.261697 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:01:43.637291 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:43.637182 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:01:43.821069 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:43.821017 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:01:47.025523 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:47.025488 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:01:47.537822 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:47.537787 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:01:47.650760 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:47.650725 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:01:49.193208 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:49.193162 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:01:50.831822 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:50.831788 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:01:50.943567 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:50.943518 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:01:52.261942 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:52.261902 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:01:52.371566 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:52.371532 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:01:52.571629 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:52.571549 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:01:52.691702 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:52.691668 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:01:53.972235 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:53.972195 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:01:55.217035 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:01:55.216990 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:01:55.599631 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:55.599529 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:01:55.784922 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:55.784884 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:01:57.286117 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:57.286013 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:01:58.879346 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:01:58.879288 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:00.496332 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:00.496294 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:02.084252 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:02.084211 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:03.784410 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:03.784336 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:05.397372 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:05.397300 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:05.749564 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:05.749529 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:06.498238 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:06.498188 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:06.649549 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:06.649505 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:07.091086 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:07.091037 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:07.962027 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:07.961953 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:09.004566 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:09.004518 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:10.308608 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:10.308568 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:02:10.701140 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:10.701024 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:12.384328 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:12.384288 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:14.080548 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:14.080498 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:15.815476 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:15.815440 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:17.682152 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:17.682113 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:19.443535 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:19.443488 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:21.023300 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:21.023266 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:22.647526 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:22.647482 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:24.291356 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:24.291320 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:25.217025 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:02:25.216985 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:02:25.893143 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:25.893054 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:27.481691 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:27.481654 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:02:27.586142 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:27.586109 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:29.191690 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:29.191655 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:30.683394 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:30.683348 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:32.387744 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:32.387703 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:33.983734 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:33.983697 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:35.574479 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:35.574422 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:37.173193 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:37.173157 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:38.780348 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:38.780309 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:40.284644 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:40.284606 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:41.942621 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:41.942535 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:43.523250 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:43.523212 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:43.792160 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:43.792063 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:45.265200 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:45.265166 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:46.827040 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:46.827005 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:48.480359 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:48.480322 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:50.236797 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:50.236738 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:51.138263 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:51.138227 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:51.658435 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:51.658399 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:51.911422 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:51.911381 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:52.561455 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:52.561407 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:02:53.592870 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:53.592834 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:55.217224 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:02:55.217181 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:02:55.258399 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:55.258353 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:56.153277 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:56.153231 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:56.790935 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:56.790893 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:57.542634 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:57.542572 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:57.667248 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:57.667211 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:57.793802 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:57.793714 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:58.474214 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:02:58.472492 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:00.149624 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:00.149590 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:01.478062 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:01.478018 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:03:06.315936 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:06.315902 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:06.440791 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:06.440757 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:07.059833 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:07.059793 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:07.518267 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:07.518231 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:21.083294 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:21.083211 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:03:25.217691 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:03:25.217653 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:03:26.540401 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:26.540369 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l"] Oct 30 03:03:26.540906 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:26.540559 5998 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l" podUID="9b4e4651-47b0-4440-b8a1-87d7fdb96f46" containerName="manager" containerID="cri-o://4f3f8d56a6896e21ec740dd58187a225dc5600a6b8043a40cc3fbd16275a7f66" gracePeriod=10 Oct 30 03:03:26.795076 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:26.795008 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l" Oct 30 03:03:26.815275 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:26.815245 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4e4651-47b0-4440-b8a1-87d7fdb96f46-kube-api-access-srp6k\" (UniqueName: \"kubernetes.io/projected/9b4e4651-47b0-4440-b8a1-87d7fdb96f46-kube-api-access-srp6k\") pod \"9b4e4651-47b0-4440-b8a1-87d7fdb96f46\" (UID: \"9b4e4651-47b0-4440-b8a1-87d7fdb96f46\") " Oct 30 03:03:26.815419 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:26.815283 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b4e4651-47b0-4440-b8a1-87d7fdb96f46-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b4e4651-47b0-4440-b8a1-87d7fdb96f46-extensions-socket-volume\") pod \"9b4e4651-47b0-4440-b8a1-87d7fdb96f46\" (UID: \"9b4e4651-47b0-4440-b8a1-87d7fdb96f46\") " Oct 30 03:03:26.815698 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:26.815675 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4e4651-47b0-4440-b8a1-87d7fdb96f46-extensions-socket-volume" pod "9b4e4651-47b0-4440-b8a1-87d7fdb96f46" (UID: "9b4e4651-47b0-4440-b8a1-87d7fdb96f46"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:03:26.817465 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:26.817445 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4e4651-47b0-4440-b8a1-87d7fdb96f46-kube-api-access-srp6k" pod "9b4e4651-47b0-4440-b8a1-87d7fdb96f46" (UID: "9b4e4651-47b0-4440-b8a1-87d7fdb96f46"). InnerVolumeSpecName "kube-api-access-srp6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:03:26.916608 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:26.916562 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srp6k\" (UniqueName: \"kubernetes.io/projected/9b4e4651-47b0-4440-b8a1-87d7fdb96f46-kube-api-access-srp6k\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:03:26.916608 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:26.916606 5998 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b4e4651-47b0-4440-b8a1-87d7fdb96f46-extensions-socket-volume\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:03:27.446718 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:27.446681 5998 generic.go:356] "Generic (PLEG): container finished" podID="9b4e4651-47b0-4440-b8a1-87d7fdb96f46" containerID="4f3f8d56a6896e21ec740dd58187a225dc5600a6b8043a40cc3fbd16275a7f66" exitCode=0 Oct 30 03:03:27.446718 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:27.446723 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l" event={"ID":"9b4e4651-47b0-4440-b8a1-87d7fdb96f46","Type":"ContainerDied","Data":"4f3f8d56a6896e21ec740dd58187a225dc5600a6b8043a40cc3fbd16275a7f66"} Oct 30 03:03:27.446950 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:27.446762 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l" event={"ID":"9b4e4651-47b0-4440-b8a1-87d7fdb96f46","Type":"ContainerDied","Data":"ee1ce8f8e19c127af652747fe997446d9cccaf4ba369f9207a446a4eae3efc04"} Oct 30 03:03:27.446950 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:27.446784 5998 scope.go:122] "RemoveContainer" containerID="4f3f8d56a6896e21ec740dd58187a225dc5600a6b8043a40cc3fbd16275a7f66" Oct 30 03:03:27.446950 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:27.446725 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l" Oct 30 03:03:27.460751 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:27.460725 5998 scope.go:122] "RemoveContainer" containerID="4f3f8d56a6896e21ec740dd58187a225dc5600a6b8043a40cc3fbd16275a7f66" Oct 30 03:03:27.461121 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:03:27.461079 5998 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f3f8d56a6896e21ec740dd58187a225dc5600a6b8043a40cc3fbd16275a7f66\": container with ID starting with 4f3f8d56a6896e21ec740dd58187a225dc5600a6b8043a40cc3fbd16275a7f66 not found: ID does not exist" containerID="4f3f8d56a6896e21ec740dd58187a225dc5600a6b8043a40cc3fbd16275a7f66" Oct 30 03:03:27.461244 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:27.461134 5998 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f3f8d56a6896e21ec740dd58187a225dc5600a6b8043a40cc3fbd16275a7f66"} err="failed to get container status \"4f3f8d56a6896e21ec740dd58187a225dc5600a6b8043a40cc3fbd16275a7f66\": rpc error: code = NotFound desc = could not find container \"4f3f8d56a6896e21ec740dd58187a225dc5600a6b8043a40cc3fbd16275a7f66\": container with ID starting with 4f3f8d56a6896e21ec740dd58187a225dc5600a6b8043a40cc3fbd16275a7f66 not found: ID does not exist" Oct 30 03:03:27.507319 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:27.507272 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l"] Oct 30 03:03:27.517668 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:27.517644 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rjj9l"] Oct 30 03:03:28.537458 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:28.537420 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b4e4651-47b0-4440-b8a1-87d7fdb96f46" path="/var/lib/kubelet/pods/9b4e4651-47b0-4440-b8a1-87d7fdb96f46/volumes" Oct 30 03:03:32.473198 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:32.473159 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:34.694803 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:34.694756 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:40.978336 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:40.978298 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:43.907619 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:03:43.907581 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:55.216059 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:03:55.216018 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:04:25.216933 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:04:25.216891 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:04:29.715191 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:29.715157 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-s228v"] Oct 30 03:04:29.715612 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:29.715549 5998 cpu_manager.go:414] "containerMap: removing container" podUID="1359a1bd-f5ae-4843-bd73-6f9b2f289f07" containerName="revoke-keys" Oct 30 03:04:29.715612 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:29.715563 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1359a1bd-f5ae-4843-bd73-6f9b2f289f07" containerName="revoke-keys" Oct 30 03:04:29.715612 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:29.715587 5998 cpu_manager.go:414] "containerMap: removing container" podUID="9b4e4651-47b0-4440-b8a1-87d7fdb96f46" containerName="manager" Oct 30 03:04:29.715612 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:29.715592 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4e4651-47b0-4440-b8a1-87d7fdb96f46" containerName="manager" Oct 30 03:04:29.715758 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:29.715661 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4e4651-47b0-4440-b8a1-87d7fdb96f46" containerName="manager" Oct 30 03:04:29.715758 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:29.715674 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="1359a1bd-f5ae-4843-bd73-6f9b2f289f07" containerName="revoke-keys" Oct 30 03:04:29.718513 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:29.718494 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-s228v" Oct 30 03:04:29.733559 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:29.733534 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v78hl\"" Oct 30 03:04:29.745754 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:29.745723 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-s228v"] Oct 30 03:04:29.807752 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:29.807704 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d3c1f40-053d-4b00-83a2-a6308a90a0b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-s228v\" (UID: \"6d3c1f40-053d-4b00-83a2-a6308a90a0b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-s228v" Oct 30 03:04:29.807934 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:29.807760 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpcrq\" (UniqueName: \"kubernetes.io/projected/6d3c1f40-053d-4b00-83a2-a6308a90a0b8-kube-api-access-fpcrq\") pod \"kuadrant-operator-controller-manager-678d87766f-s228v\" (UID: \"6d3c1f40-053d-4b00-83a2-a6308a90a0b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-s228v" Oct 30 03:04:29.908847 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:29.908807 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d3c1f40-053d-4b00-83a2-a6308a90a0b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-s228v\" (UID: \"6d3c1f40-053d-4b00-83a2-a6308a90a0b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-s228v" Oct 30 03:04:29.909036 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:29.908856 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpcrq\" (UniqueName: \"kubernetes.io/projected/6d3c1f40-053d-4b00-83a2-a6308a90a0b8-kube-api-access-fpcrq\") pod \"kuadrant-operator-controller-manager-678d87766f-s228v\" (UID: \"6d3c1f40-053d-4b00-83a2-a6308a90a0b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-s228v" Oct 30 03:04:29.909253 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:29.909229 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d3c1f40-053d-4b00-83a2-a6308a90a0b8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-s228v\" (UID: \"6d3c1f40-053d-4b00-83a2-a6308a90a0b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-s228v" Oct 30 03:04:29.930488 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:29.930455 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpcrq\" (UniqueName: \"kubernetes.io/projected/6d3c1f40-053d-4b00-83a2-a6308a90a0b8-kube-api-access-fpcrq\") pod \"kuadrant-operator-controller-manager-678d87766f-s228v\" (UID: \"6d3c1f40-053d-4b00-83a2-a6308a90a0b8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-s228v" Oct 30 03:04:30.031492 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:30.031404 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-s228v" Oct 30 03:04:30.365661 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:30.365567 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-s228v"] Oct 30 03:04:30.383649 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:04:30.383609 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3c1f40_053d_4b00_83a2_a6308a90a0b8.slice/crio-5d1639346ff6127fbc6f88bae8708908c97d308431578d272c240814bc33ceca WatchSource:0}: Error finding container 5d1639346ff6127fbc6f88bae8708908c97d308431578d272c240814bc33ceca: Status 404 returned error can't find the container with id 5d1639346ff6127fbc6f88bae8708908c97d308431578d272c240814bc33ceca Oct 30 03:04:30.767620 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:30.767579 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-s228v" event={"ID":"6d3c1f40-053d-4b00-83a2-a6308a90a0b8","Type":"ContainerStarted","Data":"e278eab524d37715bf7aaf9ab13923b3669c909a89d3616bfd99bd471cbcd584"} Oct 30 03:04:30.768047 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:30.767769 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-s228v" Oct 30 03:04:30.768047 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:30.767796 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-s228v" event={"ID":"6d3c1f40-053d-4b00-83a2-a6308a90a0b8","Type":"ContainerStarted","Data":"5d1639346ff6127fbc6f88bae8708908c97d308431578d272c240814bc33ceca"} Oct 30 03:04:30.819662 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:30.819607 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-s228v" podStartSLOduration=1.819592761 podStartE2EDuration="1.819592761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:30.818437129 +0000 UTC m=+5700.397228211" watchObservedRunningTime="2026-10-30 03:04:30.819592761 +0000 UTC m=+5700.398383839" Oct 30 03:04:37.092889 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:37.092845 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:41.774821 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:41.774787 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-s228v" Oct 30 03:04:44.065283 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:44.065223 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:49.991032 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:49.990985 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:04:55.215992 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:04:55.215949 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:04:59.295954 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:59.295910 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:59.408305 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:59.408266 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:59.791207 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:59.791168 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:59.897692 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:04:59.897659 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:05:05.374131 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:05.374065 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:05:10.056112 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:10.055353 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4"] Oct 30 03:05:10.074115 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:10.056665 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jrgm4"] Oct 30 03:05:10.574328 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:10.574287 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dbcaf5a-6527-4624-a6bc-c161cc9a0a32" path="/var/lib/kubelet/pods/3dbcaf5a-6527-4624-a6bc-c161cc9a0a32/volumes" Oct 30 03:05:10.837098 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:10.836981 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:05:15.404219 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:15.404180 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:05:25.215854 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:05:25.215816 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:05:32.369261 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:32.369201 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:05:39.541718 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:39.541676 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8"] Oct 30 03:05:39.545762 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:39.545734 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8" Oct 30 03:05:39.560057 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:39.560028 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:05:39.571699 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:39.571673 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:05:39.571963 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:39.571939 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:05:39.572070 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:39.572025 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:05:39.573999 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:39.573976 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8"] Oct 30 03:05:39.678917 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:39.678880 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f782de00-fvvm8\" (UID: \"48ab4fe3-71c1-4d24-93a9-6c0686ef559b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8" Oct 30 03:05:39.679129 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:39.678955 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f782de00-fvvm8\" (UID: \"48ab4fe3-71c1-4d24-93a9-6c0686ef559b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8" Oct 30 03:05:39.679129 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:39.679038 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg7hx\" (UniqueName: \"kubernetes.io/projected/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-kube-api-access-kg7hx\") pod \"maas-api-revoke-sub-f782de00-fvvm8\" (UID: \"48ab4fe3-71c1-4d24-93a9-6c0686ef559b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8" Oct 30 03:05:39.780265 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:39.780227 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f782de00-fvvm8\" (UID: \"48ab4fe3-71c1-4d24-93a9-6c0686ef559b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8" Oct 30 03:05:39.780432 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:39.780298 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f782de00-fvvm8\" (UID: \"48ab4fe3-71c1-4d24-93a9-6c0686ef559b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8" Oct 30 03:05:39.780432 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:39.780327 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg7hx\" (UniqueName: \"kubernetes.io/projected/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-kube-api-access-kg7hx\") pod \"maas-api-revoke-sub-f782de00-fvvm8\" (UID: \"48ab4fe3-71c1-4d24-93a9-6c0686ef559b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8" Oct 30 03:05:39.781074 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:39.781052 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f782de00-fvvm8\" (UID: \"48ab4fe3-71c1-4d24-93a9-6c0686ef559b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8" Oct 30 03:05:39.782733 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:39.782711 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f782de00-fvvm8\" (UID: \"48ab4fe3-71c1-4d24-93a9-6c0686ef559b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8" Oct 30 03:05:39.803857 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:39.803799 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg7hx\" (UniqueName: \"kubernetes.io/projected/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-kube-api-access-kg7hx\") pod \"maas-api-revoke-sub-f782de00-fvvm8\" (UID: \"48ab4fe3-71c1-4d24-93a9-6c0686ef559b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8" Oct 30 03:05:39.858691 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:39.858661 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8" Oct 30 03:05:40.157194 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:40.157107 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8"] Oct 30 03:05:40.161486 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:05:40.161456 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ab4fe3_71c1_4d24_93a9_6c0686ef559b.slice/crio-0fc0ad17bc41628b273b1b7068a8dcbdebf97223465720cd748c709de191c472 WatchSource:0}: Error finding container 0fc0ad17bc41628b273b1b7068a8dcbdebf97223465720cd748c709de191c472: Status 404 returned error can't find the container with id 0fc0ad17bc41628b273b1b7068a8dcbdebf97223465720cd748c709de191c472 Oct 30 03:05:40.190014 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:40.189983 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8" event={"ID":"48ab4fe3-71c1-4d24-93a9-6c0686ef559b","Type":"ContainerStarted","Data":"0fc0ad17bc41628b273b1b7068a8dcbdebf97223465720cd748c709de191c472"} Oct 30 03:05:41.198112 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:41.197989 5998 generic.go:356] "Generic (PLEG): container finished" podID="48ab4fe3-71c1-4d24-93a9-6c0686ef559b" containerID="1a95dc8ef51b16084ae4f0a0b132eced0eb8b429fbce1b241721c4576261cd1d" exitCode=0 Oct 30 03:05:41.198112 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:41.198074 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8" event={"ID":"48ab4fe3-71c1-4d24-93a9-6c0686ef559b","Type":"ContainerDied","Data":"1a95dc8ef51b16084ae4f0a0b132eced0eb8b429fbce1b241721c4576261cd1d"} Oct 30 03:05:42.342388 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:42.342358 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8" Oct 30 03:05:42.504807 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:42.504723 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-maas-api-xks-ca\") pod \"48ab4fe3-71c1-4d24-93a9-6c0686ef559b\" (UID: \"48ab4fe3-71c1-4d24-93a9-6c0686ef559b\") " Oct 30 03:05:42.504953 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:42.504809 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-maas-api-openshift-ca\") pod \"48ab4fe3-71c1-4d24-93a9-6c0686ef559b\" (UID: \"48ab4fe3-71c1-4d24-93a9-6c0686ef559b\") " Oct 30 03:05:42.504953 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:42.504865 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-kube-api-access-kg7hx\" (UniqueName: \"kubernetes.io/projected/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-kube-api-access-kg7hx\") pod \"48ab4fe3-71c1-4d24-93a9-6c0686ef559b\" (UID: \"48ab4fe3-71c1-4d24-93a9-6c0686ef559b\") " Oct 30 03:05:42.505202 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:42.505179 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-maas-api-openshift-ca" pod "48ab4fe3-71c1-4d24-93a9-6c0686ef559b" (UID: "48ab4fe3-71c1-4d24-93a9-6c0686ef559b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:05:42.507002 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:42.506967 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-maas-api-xks-ca" pod "48ab4fe3-71c1-4d24-93a9-6c0686ef559b" (UID: "48ab4fe3-71c1-4d24-93a9-6c0686ef559b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:05:42.507159 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:42.506974 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-kube-api-access-kg7hx" pod "48ab4fe3-71c1-4d24-93a9-6c0686ef559b" (UID: "48ab4fe3-71c1-4d24-93a9-6c0686ef559b"). InnerVolumeSpecName "kube-api-access-kg7hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:05:42.605781 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:42.605744 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg7hx\" (UniqueName: \"kubernetes.io/projected/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-kube-api-access-kg7hx\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:05:42.605781 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:42.605777 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:05:42.605781 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:42.605787 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48ab4fe3-71c1-4d24-93a9-6c0686ef559b-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:05:43.209350 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:43.209321 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8" Oct 30 03:05:43.209350 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:43.209338 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8" event={"ID":"48ab4fe3-71c1-4d24-93a9-6c0686ef559b","Type":"ContainerDied","Data":"0fc0ad17bc41628b273b1b7068a8dcbdebf97223465720cd748c709de191c472"} Oct 30 03:05:43.209587 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:43.209385 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fc0ad17bc41628b273b1b7068a8dcbdebf97223465720cd748c709de191c472" Oct 30 03:05:44.203286 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:44.203246 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:05:49.871784 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:49.871745 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr"] Oct 30 03:05:49.872908 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:49.872874 5998 cpu_manager.go:414] "containerMap: removing container" podUID="48ab4fe3-71c1-4d24-93a9-6c0686ef559b" containerName="revoke-keys" Oct 30 03:05:49.872908 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:49.872908 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ab4fe3-71c1-4d24-93a9-6c0686ef559b" containerName="revoke-keys" Oct 30 03:05:49.873131 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:49.873042 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ab4fe3-71c1-4d24-93a9-6c0686ef559b" containerName="revoke-keys" Oct 30 03:05:49.875918 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:49.875893 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr" Oct 30 03:05:49.890401 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:49.890375 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:05:49.901984 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:49.901952 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:05:49.902178 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:49.902077 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:05:49.902271 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:49.902186 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:05:49.904716 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:49.904688 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr"] Oct 30 03:05:49.973696 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:49.973656 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7ebac2a-738c-4851-904d-c752b06d5501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34ae66f0-rpxnr\" (UID: \"d7ebac2a-738c-4851-904d-c752b06d5501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr" Oct 30 03:05:49.973696 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:49.973698 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxbsg\" (UniqueName: \"kubernetes.io/projected/d7ebac2a-738c-4851-904d-c752b06d5501-kube-api-access-fxbsg\") pod \"maas-api-revoke-sub-34ae66f0-rpxnr\" (UID: \"d7ebac2a-738c-4851-904d-c752b06d5501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr" Oct 30 03:05:49.973944 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:49.973723 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7ebac2a-738c-4851-904d-c752b06d5501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34ae66f0-rpxnr\" (UID: \"d7ebac2a-738c-4851-904d-c752b06d5501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr" Oct 30 03:05:50.074526 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:50.074487 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7ebac2a-738c-4851-904d-c752b06d5501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34ae66f0-rpxnr\" (UID: \"d7ebac2a-738c-4851-904d-c752b06d5501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr" Oct 30 03:05:50.074715 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:50.074531 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxbsg\" (UniqueName: \"kubernetes.io/projected/d7ebac2a-738c-4851-904d-c752b06d5501-kube-api-access-fxbsg\") pod \"maas-api-revoke-sub-34ae66f0-rpxnr\" (UID: \"d7ebac2a-738c-4851-904d-c752b06d5501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr" Oct 30 03:05:50.074715 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:50.074559 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7ebac2a-738c-4851-904d-c752b06d5501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34ae66f0-rpxnr\" (UID: \"d7ebac2a-738c-4851-904d-c752b06d5501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr" Oct 30 03:05:50.075291 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:50.075259 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7ebac2a-738c-4851-904d-c752b06d5501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34ae66f0-rpxnr\" (UID: \"d7ebac2a-738c-4851-904d-c752b06d5501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr" Oct 30 03:05:50.077033 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:50.077009 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7ebac2a-738c-4851-904d-c752b06d5501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34ae66f0-rpxnr\" (UID: \"d7ebac2a-738c-4851-904d-c752b06d5501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr" Oct 30 03:05:50.097387 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:50.097358 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxbsg\" (UniqueName: \"kubernetes.io/projected/d7ebac2a-738c-4851-904d-c752b06d5501-kube-api-access-fxbsg\") pod \"maas-api-revoke-sub-34ae66f0-rpxnr\" (UID: \"d7ebac2a-738c-4851-904d-c752b06d5501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr" Oct 30 03:05:50.189242 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:50.189207 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr" Oct 30 03:05:50.496185 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:50.496077 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr"] Oct 30 03:05:50.501228 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:05:50.501185 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ebac2a_738c_4851_904d_c752b06d5501.slice/crio-a1031e5d90bc4f82ecbfaa3bb70a9a6aa03c616ede18bbb97dfaa0512f2e4c52 WatchSource:0}: Error finding container a1031e5d90bc4f82ecbfaa3bb70a9a6aa03c616ede18bbb97dfaa0512f2e4c52: Status 404 returned error can't find the container with id a1031e5d90bc4f82ecbfaa3bb70a9a6aa03c616ede18bbb97dfaa0512f2e4c52 Oct 30 03:05:51.252939 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:51.252898 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr" event={"ID":"d7ebac2a-738c-4851-904d-c752b06d5501","Type":"ContainerStarted","Data":"80bac8b7ce90a650ec22913be602647fc2750d53157bc7c09f8e8bff9c324127"} Oct 30 03:05:51.253339 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:51.252966 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr" event={"ID":"d7ebac2a-738c-4851-904d-c752b06d5501","Type":"ContainerStarted","Data":"a1031e5d90bc4f82ecbfaa3bb70a9a6aa03c616ede18bbb97dfaa0512f2e4c52"} Oct 30 03:05:51.294216 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:51.294160 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr" podStartSLOduration=2.294145554 podStartE2EDuration="2.294145554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:51.293280433 +0000 UTC m=+5780.872071513" watchObservedRunningTime="2026-10-30 03:05:51.294145554 +0000 UTC m=+5780.872936632" Oct 30 03:05:52.260266 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:52.260225 5998 generic.go:356] "Generic (PLEG): container finished" podID="d7ebac2a-738c-4851-904d-c752b06d5501" containerID="80bac8b7ce90a650ec22913be602647fc2750d53157bc7c09f8e8bff9c324127" exitCode=0 Oct 30 03:05:52.260746 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:52.260310 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr" event={"ID":"d7ebac2a-738c-4851-904d-c752b06d5501","Type":"ContainerDied","Data":"80bac8b7ce90a650ec22913be602647fc2750d53157bc7c09f8e8bff9c324127"} Oct 30 03:05:53.401436 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:53.401402 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr" Oct 30 03:05:53.508201 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:53.508161 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ebac2a-738c-4851-904d-c752b06d5501-kube-api-access-fxbsg\" (UniqueName: \"kubernetes.io/projected/d7ebac2a-738c-4851-904d-c752b06d5501-kube-api-access-fxbsg\") pod \"d7ebac2a-738c-4851-904d-c752b06d5501\" (UID: \"d7ebac2a-738c-4851-904d-c752b06d5501\") " Oct 30 03:05:53.508392 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:53.508212 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7ebac2a-738c-4851-904d-c752b06d5501-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7ebac2a-738c-4851-904d-c752b06d5501-maas-api-openshift-ca\") pod \"d7ebac2a-738c-4851-904d-c752b06d5501\" (UID: \"d7ebac2a-738c-4851-904d-c752b06d5501\") " Oct 30 03:05:53.508392 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:53.508244 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7ebac2a-738c-4851-904d-c752b06d5501-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7ebac2a-738c-4851-904d-c752b06d5501-maas-api-xks-ca\") pod \"d7ebac2a-738c-4851-904d-c752b06d5501\" (UID: \"d7ebac2a-738c-4851-904d-c752b06d5501\") " Oct 30 03:05:53.508620 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:53.508599 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ebac2a-738c-4851-904d-c752b06d5501-maas-api-openshift-ca" pod "d7ebac2a-738c-4851-904d-c752b06d5501" (UID: "d7ebac2a-738c-4851-904d-c752b06d5501"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:05:53.510510 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:53.510484 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ebac2a-738c-4851-904d-c752b06d5501-kube-api-access-fxbsg" pod "d7ebac2a-738c-4851-904d-c752b06d5501" (UID: "d7ebac2a-738c-4851-904d-c752b06d5501"). InnerVolumeSpecName "kube-api-access-fxbsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:05:53.510645 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:53.510481 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7ebac2a-738c-4851-904d-c752b06d5501-maas-api-xks-ca" pod "d7ebac2a-738c-4851-904d-c752b06d5501" (UID: "d7ebac2a-738c-4851-904d-c752b06d5501"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:05:53.609422 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:53.609386 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7ebac2a-738c-4851-904d-c752b06d5501-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:05:53.609422 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:53.609417 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7ebac2a-738c-4851-904d-c752b06d5501-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:05:53.609422 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:53.609427 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxbsg\" (UniqueName: \"kubernetes.io/projected/d7ebac2a-738c-4851-904d-c752b06d5501-kube-api-access-fxbsg\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:05:54.271490 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:54.271459 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr" Oct 30 03:05:54.271685 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:54.271505 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr" event={"ID":"d7ebac2a-738c-4851-904d-c752b06d5501","Type":"ContainerDied","Data":"a1031e5d90bc4f82ecbfaa3bb70a9a6aa03c616ede18bbb97dfaa0512f2e4c52"} Oct 30 03:05:54.271685 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:05:54.271556 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1031e5d90bc4f82ecbfaa3bb70a9a6aa03c616ede18bbb97dfaa0512f2e4c52" Oct 30 03:05:55.215216 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:05:55.215172 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:06:01.676804 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:01.676763 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:06:06.590716 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:06.590671 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:06:12.409018 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.408976 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69"] Oct 30 03:06:12.409447 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.409429 5998 cpu_manager.go:414] "containerMap: removing container" podUID="d7ebac2a-738c-4851-904d-c752b06d5501" containerName="revoke-keys" Oct 30 03:06:12.409447 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.409443 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7ebac2a-738c-4851-904d-c752b06d5501" containerName="revoke-keys" Oct 30 03:06:12.409554 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.409543 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7ebac2a-738c-4851-904d-c752b06d5501" containerName="revoke-keys" Oct 30 03:06:12.414014 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.413997 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69" Oct 30 03:06:12.428878 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.428851 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:06:12.440327 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.440291 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:06:12.440473 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.440335 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:06:12.440699 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.440681 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:06:12.442816 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.442797 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69"] Oct 30 03:06:12.523189 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.523153 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac2f8859-l2b69\" (UID: \"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69" Oct 30 03:06:12.523391 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.523307 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac2f8859-l2b69\" (UID: \"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69" Oct 30 03:06:12.523471 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.523391 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8jss\" (UniqueName: \"kubernetes.io/projected/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-kube-api-access-g8jss\") pod \"maas-api-revoke-sub-ac2f8859-l2b69\" (UID: \"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69" Oct 30 03:06:12.624190 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.624149 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac2f8859-l2b69\" (UID: \"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69" Oct 30 03:06:12.624398 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.624204 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8jss\" (UniqueName: \"kubernetes.io/projected/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-kube-api-access-g8jss\") pod \"maas-api-revoke-sub-ac2f8859-l2b69\" (UID: \"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69" Oct 30 03:06:12.624398 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.624274 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac2f8859-l2b69\" (UID: \"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69" Oct 30 03:06:12.625071 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.625042 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac2f8859-l2b69\" (UID: \"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69" Oct 30 03:06:12.627055 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.627022 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac2f8859-l2b69\" (UID: \"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69" Oct 30 03:06:12.647063 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.647040 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8jss\" (UniqueName: \"kubernetes.io/projected/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-kube-api-access-g8jss\") pod \"maas-api-revoke-sub-ac2f8859-l2b69\" (UID: \"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69" Oct 30 03:06:12.727168 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:12.727129 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69" Oct 30 03:06:13.036667 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:13.036569 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69"] Oct 30 03:06:13.040971 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:06:13.040933 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3fac7d0_7f99_4b09_bbbc_3b32497a67ee.slice/crio-5a58b5166442b1332e0f23ddab3a187147bcce908d1cf0a02420e046d87bc7d6 WatchSource:0}: Error finding container 5a58b5166442b1332e0f23ddab3a187147bcce908d1cf0a02420e046d87bc7d6: Status 404 returned error can't find the container with id 5a58b5166442b1332e0f23ddab3a187147bcce908d1cf0a02420e046d87bc7d6 Oct 30 03:06:13.393525 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:13.393478 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69" event={"ID":"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee","Type":"ContainerStarted","Data":"96cae6e8db388c59c21fe81fae17e3d239e2ca72383872e04538c9550988924f"} Oct 30 03:06:13.393725 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:13.393534 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69" event={"ID":"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee","Type":"ContainerStarted","Data":"5a58b5166442b1332e0f23ddab3a187147bcce908d1cf0a02420e046d87bc7d6"} Oct 30 03:06:13.436303 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:13.436244 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69" podStartSLOduration=1.436229617 podStartE2EDuration="1.436229617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:13.434417906 +0000 UTC m=+5803.013208967" watchObservedRunningTime="2026-10-30 03:06:13.436229617 +0000 UTC m=+5803.015020698" Oct 30 03:06:14.400238 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:14.400202 5998 generic.go:356] "Generic (PLEG): container finished" podID="f3fac7d0-7f99-4b09-bbbc-3b32497a67ee" containerID="96cae6e8db388c59c21fe81fae17e3d239e2ca72383872e04538c9550988924f" exitCode=0 Oct 30 03:06:14.400426 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:14.400246 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69" event={"ID":"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee","Type":"ContainerDied","Data":"96cae6e8db388c59c21fe81fae17e3d239e2ca72383872e04538c9550988924f"} Oct 30 03:06:15.540373 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:15.540346 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69" Oct 30 03:06:15.653195 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:15.653158 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-maas-api-openshift-ca\") pod \"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee\" (UID: \"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee\") " Oct 30 03:06:15.653387 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:15.653227 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-kube-api-access-g8jss\" (UniqueName: \"kubernetes.io/projected/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-kube-api-access-g8jss\") pod \"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee\" (UID: \"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee\") " Oct 30 03:06:15.653387 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:15.653287 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-maas-api-xks-ca\") pod \"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee\" (UID: \"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee\") " Oct 30 03:06:15.653550 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:15.653524 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-maas-api-openshift-ca" pod "f3fac7d0-7f99-4b09-bbbc-3b32497a67ee" (UID: "f3fac7d0-7f99-4b09-bbbc-3b32497a67ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:06:15.655373 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:15.655340 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-maas-api-xks-ca" pod "f3fac7d0-7f99-4b09-bbbc-3b32497a67ee" (UID: "f3fac7d0-7f99-4b09-bbbc-3b32497a67ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:06:15.655373 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:15.655358 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-kube-api-access-g8jss" pod "f3fac7d0-7f99-4b09-bbbc-3b32497a67ee" (UID: "f3fac7d0-7f99-4b09-bbbc-3b32497a67ee"). InnerVolumeSpecName "kube-api-access-g8jss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:06:15.754630 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:15.754589 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:06:15.754630 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:15.754625 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:06:15.754630 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:15.754637 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8jss\" (UniqueName: \"kubernetes.io/projected/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee-kube-api-access-g8jss\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:06:16.411383 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:16.411338 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69" event={"ID":"f3fac7d0-7f99-4b09-bbbc-3b32497a67ee","Type":"ContainerDied","Data":"5a58b5166442b1332e0f23ddab3a187147bcce908d1cf0a02420e046d87bc7d6"} Oct 30 03:06:16.411579 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:16.411359 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69" Oct 30 03:06:16.411579 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:16.411432 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a58b5166442b1332e0f23ddab3a187147bcce908d1cf0a02420e046d87bc7d6" Oct 30 03:06:23.759106 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:23.759045 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:06:24.209250 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:24.209187 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:06:24.315648 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:24.315569 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:06:25.220816 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:06:25.220773 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:06:28.590072 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:28.590026 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:06:31.008116 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:31.005223 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-ppvtr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:06:31.008116 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:31.005320 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-ppvtr" podUID="961683fb-469d-4d4a-9463-269eb264251a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:06:31.473282 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:31.473235 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-28ssd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:06:31.473637 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:31.473598 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" podUID="d8da8b22-5586-419c-9378-03d3d2d2c47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:06:35.181600 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:35.181553 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:06:35.549689 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:35.549592 5998 scope.go:122] "RemoveContainer" containerID="0ea42d9f36d61d9f6c79efb33ef0d3cf239c4e9ee7684d94e869d027b9744c30" Oct 30 03:06:35.558756 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:35.558727 5998 scope.go:122] "RemoveContainer" containerID="9533f30dc5dd5ea12d16c699085c56beda3afa8688d093feec5d907706e62794" Oct 30 03:06:43.315066 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.315011 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2"] Oct 30 03:06:43.315598 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.315499 5998 cpu_manager.go:414] "containerMap: removing container" podUID="f3fac7d0-7f99-4b09-bbbc-3b32497a67ee" containerName="revoke-keys" Oct 30 03:06:43.315598 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.315518 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3fac7d0-7f99-4b09-bbbc-3b32497a67ee" containerName="revoke-keys" Oct 30 03:06:43.315683 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.315616 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3fac7d0-7f99-4b09-bbbc-3b32497a67ee" containerName="revoke-keys" Oct 30 03:06:43.320343 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.320323 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2" Oct 30 03:06:43.339951 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.339918 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2"] Oct 30 03:06:43.349464 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.349431 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:06:43.349464 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.349434 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:06:43.349706 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.349434 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:06:43.349706 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.349507 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:06:43.457859 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.457820 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtcjb\" (UniqueName: \"kubernetes.io/projected/c6d8e235-ccbe-420e-b07b-5c3f96252538-kube-api-access-jtcjb\") pod \"maas-api-revoke-sub-4ff3b3d0-bkzz2\" (UID: \"c6d8e235-ccbe-420e-b07b-5c3f96252538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2" Oct 30 03:06:43.458073 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.457955 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d8e235-ccbe-420e-b07b-5c3f96252538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ff3b3d0-bkzz2\" (UID: \"c6d8e235-ccbe-420e-b07b-5c3f96252538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2" Oct 30 03:06:43.458182 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.458113 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d8e235-ccbe-420e-b07b-5c3f96252538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ff3b3d0-bkzz2\" (UID: \"c6d8e235-ccbe-420e-b07b-5c3f96252538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2" Oct 30 03:06:43.558593 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.558547 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d8e235-ccbe-420e-b07b-5c3f96252538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ff3b3d0-bkzz2\" (UID: \"c6d8e235-ccbe-420e-b07b-5c3f96252538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2" Oct 30 03:06:43.558811 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.558637 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d8e235-ccbe-420e-b07b-5c3f96252538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ff3b3d0-bkzz2\" (UID: \"c6d8e235-ccbe-420e-b07b-5c3f96252538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2" Oct 30 03:06:43.558811 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.558679 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtcjb\" (UniqueName: \"kubernetes.io/projected/c6d8e235-ccbe-420e-b07b-5c3f96252538-kube-api-access-jtcjb\") pod \"maas-api-revoke-sub-4ff3b3d0-bkzz2\" (UID: \"c6d8e235-ccbe-420e-b07b-5c3f96252538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2" Oct 30 03:06:43.559282 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.559258 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d8e235-ccbe-420e-b07b-5c3f96252538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ff3b3d0-bkzz2\" (UID: \"c6d8e235-ccbe-420e-b07b-5c3f96252538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2" Oct 30 03:06:43.560976 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.560949 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d8e235-ccbe-420e-b07b-5c3f96252538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ff3b3d0-bkzz2\" (UID: \"c6d8e235-ccbe-420e-b07b-5c3f96252538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2" Oct 30 03:06:43.581738 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.581667 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtcjb\" (UniqueName: \"kubernetes.io/projected/c6d8e235-ccbe-420e-b07b-5c3f96252538-kube-api-access-jtcjb\") pod \"maas-api-revoke-sub-4ff3b3d0-bkzz2\" (UID: \"c6d8e235-ccbe-420e-b07b-5c3f96252538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2" Oct 30 03:06:43.632905 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.632872 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2" Oct 30 03:06:43.933012 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:43.932969 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2"] Oct 30 03:06:43.937824 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:06:43.937787 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d8e235_ccbe_420e_b07b_5c3f96252538.slice/crio-66d297a365ef9384440e46b5e25c669d8f8a49aca1427103957282555265cf34 WatchSource:0}: Error finding container 66d297a365ef9384440e46b5e25c669d8f8a49aca1427103957282555265cf34: Status 404 returned error can't find the container with id 66d297a365ef9384440e46b5e25c669d8f8a49aca1427103957282555265cf34 Oct 30 03:06:44.296909 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:44.296871 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:06:44.610070 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:44.609967 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2" event={"ID":"c6d8e235-ccbe-420e-b07b-5c3f96252538","Type":"ContainerStarted","Data":"ebaff71ee5d96276bf9e98cb17e85863fca40229c31093f9a9148781d3e3bf9e"} Oct 30 03:06:44.610070 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:44.610037 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2" event={"ID":"c6d8e235-ccbe-420e-b07b-5c3f96252538","Type":"ContainerStarted","Data":"66d297a365ef9384440e46b5e25c669d8f8a49aca1427103957282555265cf34"} Oct 30 03:06:44.649880 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:44.649819 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2" podStartSLOduration=1.649803875 podStartE2EDuration="1.649803875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:44.648824445 +0000 UTC m=+5834.227615526" watchObservedRunningTime="2026-10-30 03:06:44.649803875 +0000 UTC m=+5834.228594949" Oct 30 03:06:45.617609 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:45.617572 5998 generic.go:356] "Generic (PLEG): container finished" podID="c6d8e235-ccbe-420e-b07b-5c3f96252538" containerID="ebaff71ee5d96276bf9e98cb17e85863fca40229c31093f9a9148781d3e3bf9e" exitCode=0 Oct 30 03:06:45.618023 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:45.617666 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2" event={"ID":"c6d8e235-ccbe-420e-b07b-5c3f96252538","Type":"ContainerDied","Data":"ebaff71ee5d96276bf9e98cb17e85863fca40229c31093f9a9148781d3e3bf9e"} Oct 30 03:06:46.758594 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:46.758564 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2" Oct 30 03:06:46.891004 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:46.890901 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d8e235-ccbe-420e-b07b-5c3f96252538-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d8e235-ccbe-420e-b07b-5c3f96252538-maas-api-openshift-ca\") pod \"c6d8e235-ccbe-420e-b07b-5c3f96252538\" (UID: \"c6d8e235-ccbe-420e-b07b-5c3f96252538\") " Oct 30 03:06:46.891004 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:46.890953 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d8e235-ccbe-420e-b07b-5c3f96252538-kube-api-access-jtcjb\" (UniqueName: \"kubernetes.io/projected/c6d8e235-ccbe-420e-b07b-5c3f96252538-kube-api-access-jtcjb\") pod \"c6d8e235-ccbe-420e-b07b-5c3f96252538\" (UID: \"c6d8e235-ccbe-420e-b07b-5c3f96252538\") " Oct 30 03:06:46.891004 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:46.891003 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6d8e235-ccbe-420e-b07b-5c3f96252538-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d8e235-ccbe-420e-b07b-5c3f96252538-maas-api-xks-ca\") pod \"c6d8e235-ccbe-420e-b07b-5c3f96252538\" (UID: \"c6d8e235-ccbe-420e-b07b-5c3f96252538\") " Oct 30 03:06:46.891373 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:46.891340 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d8e235-ccbe-420e-b07b-5c3f96252538-maas-api-openshift-ca" pod "c6d8e235-ccbe-420e-b07b-5c3f96252538" (UID: "c6d8e235-ccbe-420e-b07b-5c3f96252538"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:06:46.893286 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:46.893255 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d8e235-ccbe-420e-b07b-5c3f96252538-maas-api-xks-ca" pod "c6d8e235-ccbe-420e-b07b-5c3f96252538" (UID: "c6d8e235-ccbe-420e-b07b-5c3f96252538"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:06:46.893286 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:46.893270 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d8e235-ccbe-420e-b07b-5c3f96252538-kube-api-access-jtcjb" pod "c6d8e235-ccbe-420e-b07b-5c3f96252538" (UID: "c6d8e235-ccbe-420e-b07b-5c3f96252538"). InnerVolumeSpecName "kube-api-access-jtcjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:06:46.992388 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:46.992340 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6d8e235-ccbe-420e-b07b-5c3f96252538-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:06:46.992388 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:46.992378 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtcjb\" (UniqueName: \"kubernetes.io/projected/c6d8e235-ccbe-420e-b07b-5c3f96252538-kube-api-access-jtcjb\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:06:46.992388 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:46.992389 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6d8e235-ccbe-420e-b07b-5c3f96252538-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:06:47.629231 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:47.629198 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2" Oct 30 03:06:47.629455 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:47.629229 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff3b3d0-bkzz2" event={"ID":"c6d8e235-ccbe-420e-b07b-5c3f96252538","Type":"ContainerDied","Data":"66d297a365ef9384440e46b5e25c669d8f8a49aca1427103957282555265cf34"} Oct 30 03:06:47.629455 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:47.629281 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66d297a365ef9384440e46b5e25c669d8f8a49aca1427103957282555265cf34" Oct 30 03:06:53.723566 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.723510 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp"] Oct 30 03:06:53.724262 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.724063 5998 cpu_manager.go:414] "containerMap: removing container" podUID="c6d8e235-ccbe-420e-b07b-5c3f96252538" containerName="revoke-keys" Oct 30 03:06:53.724262 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.724084 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d8e235-ccbe-420e-b07b-5c3f96252538" containerName="revoke-keys" Oct 30 03:06:53.724262 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.724242 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d8e235-ccbe-420e-b07b-5c3f96252538" containerName="revoke-keys" Oct 30 03:06:53.727278 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.727254 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp" Oct 30 03:06:53.741751 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.741718 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:06:53.741921 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.741872 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:06:53.753499 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.753469 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:06:53.762427 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.754457 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:06:53.762427 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.757460 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp"] Oct 30 03:06:53.859827 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.859787 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1ce96eb-v7fhp\" (UID: \"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp" Oct 30 03:06:53.859827 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.859829 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml5qt\" (UniqueName: \"kubernetes.io/projected/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-kube-api-access-ml5qt\") pod \"maas-api-revoke-sub-b1ce96eb-v7fhp\" (UID: \"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp" Oct 30 03:06:53.860051 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.859955 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1ce96eb-v7fhp\" (UID: \"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp" Oct 30 03:06:53.961470 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.961425 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1ce96eb-v7fhp\" (UID: \"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp" Oct 30 03:06:53.961691 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.961476 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml5qt\" (UniqueName: \"kubernetes.io/projected/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-kube-api-access-ml5qt\") pod \"maas-api-revoke-sub-b1ce96eb-v7fhp\" (UID: \"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp" Oct 30 03:06:53.961691 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.961553 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1ce96eb-v7fhp\" (UID: \"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp" Oct 30 03:06:53.962254 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.962230 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1ce96eb-v7fhp\" (UID: \"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp" Oct 30 03:06:53.964163 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.964133 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1ce96eb-v7fhp\" (UID: \"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp" Oct 30 03:06:53.984261 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:53.984194 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml5qt\" (UniqueName: \"kubernetes.io/projected/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-kube-api-access-ml5qt\") pod \"maas-api-revoke-sub-b1ce96eb-v7fhp\" (UID: \"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp" Oct 30 03:06:54.040857 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:54.040819 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp" Oct 30 03:06:54.345257 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:54.345155 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp"] Oct 30 03:06:54.348715 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:06:54.348674 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd6af8d_fd7f_4715_bcf9_f3eab66a1f90.slice/crio-eacd8c6d5cdbce7c31e909a49e7063f77b2fc19d1242da3247976656f74eb8b5 WatchSource:0}: Error finding container eacd8c6d5cdbce7c31e909a49e7063f77b2fc19d1242da3247976656f74eb8b5: Status 404 returned error can't find the container with id eacd8c6d5cdbce7c31e909a49e7063f77b2fc19d1242da3247976656f74eb8b5 Oct 30 03:06:54.675661 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:54.675620 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp" event={"ID":"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90","Type":"ContainerStarted","Data":"66bfedf0ebd6b99ef51961068e8bcdcd1cabcdad1d9ba66426f4d3b31fe6f99a"} Oct 30 03:06:54.675852 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:54.675700 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp" event={"ID":"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90","Type":"ContainerStarted","Data":"eacd8c6d5cdbce7c31e909a49e7063f77b2fc19d1242da3247976656f74eb8b5"} Oct 30 03:06:54.718584 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:54.718546 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:06:54.726349 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:54.726295 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp" podStartSLOduration=1.726281765 podStartE2EDuration="1.726281765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:54.723850423 +0000 UTC m=+5844.302641504" watchObservedRunningTime="2026-10-30 03:06:54.726281765 +0000 UTC m=+5844.305072858" Oct 30 03:06:55.216942 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:06:55.216896 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:06:55.683305 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:55.683269 5998 generic.go:356] "Generic (PLEG): container finished" podID="9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90" containerID="66bfedf0ebd6b99ef51961068e8bcdcd1cabcdad1d9ba66426f4d3b31fe6f99a" exitCode=0 Oct 30 03:06:55.683498 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:55.683348 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp" event={"ID":"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90","Type":"ContainerDied","Data":"66bfedf0ebd6b99ef51961068e8bcdcd1cabcdad1d9ba66426f4d3b31fe6f99a"} Oct 30 03:06:56.843011 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:56.842982 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp" Oct 30 03:06:56.995339 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:56.995251 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-maas-api-xks-ca\") pod \"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90\" (UID: \"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90\") " Oct 30 03:06:56.995339 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:56.995297 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-maas-api-openshift-ca\") pod \"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90\" (UID: \"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90\") " Oct 30 03:06:56.995597 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:56.995418 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-kube-api-access-ml5qt\" (UniqueName: \"kubernetes.io/projected/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-kube-api-access-ml5qt\") pod \"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90\" (UID: \"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90\") " Oct 30 03:06:56.995684 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:56.995659 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-maas-api-openshift-ca" pod "9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90" (UID: "9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:06:56.997439 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:56.997411 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-maas-api-xks-ca" pod "9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90" (UID: "9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:06:56.997566 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:56.997542 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-kube-api-access-ml5qt" pod "9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90" (UID: "9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90"). InnerVolumeSpecName "kube-api-access-ml5qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:06:57.096353 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:57.096310 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml5qt\" (UniqueName: \"kubernetes.io/projected/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-kube-api-access-ml5qt\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:06:57.096353 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:57.096347 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:06:57.096353 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:57.096357 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:06:57.693913 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:57.693875 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp" Oct 30 03:06:57.694075 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:57.693884 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ce96eb-v7fhp" event={"ID":"9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90","Type":"ContainerDied","Data":"eacd8c6d5cdbce7c31e909a49e7063f77b2fc19d1242da3247976656f74eb8b5"} Oct 30 03:06:57.694075 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:06:57.694007 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eacd8c6d5cdbce7c31e909a49e7063f77b2fc19d1242da3247976656f74eb8b5" Oct 30 03:07:05.166649 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:05.166605 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:07:11.413888 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:11.413847 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:07:19.165643 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:19.165602 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:07:25.215988 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:07:25.215948 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:07:26.204235 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.204195 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc"] Oct 30 03:07:26.204687 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.204670 5998 cpu_manager.go:414] "containerMap: removing container" podUID="9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90" containerName="revoke-keys" Oct 30 03:07:26.204737 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.204689 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90" containerName="revoke-keys" Oct 30 03:07:26.204778 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.204761 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90" containerName="revoke-keys" Oct 30 03:07:26.207796 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.207771 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc" Oct 30 03:07:26.229479 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.229449 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:07:26.229862 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.229506 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:07:26.241263 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.241229 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:07:26.241539 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.241343 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:07:26.243371 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.243343 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc"] Oct 30 03:07:26.248011 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.247987 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2126872e-0a4b-4d18-8985-d1aeb474052b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bde2c551-gs8hc\" (UID: \"2126872e-0a4b-4d18-8985-d1aeb474052b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc" Oct 30 03:07:26.248161 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.248023 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2126872e-0a4b-4d18-8985-d1aeb474052b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bde2c551-gs8hc\" (UID: \"2126872e-0a4b-4d18-8985-d1aeb474052b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc" Oct 30 03:07:26.248239 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.248170 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl9ff\" (UniqueName: \"kubernetes.io/projected/2126872e-0a4b-4d18-8985-d1aeb474052b-kube-api-access-jl9ff\") pod \"maas-api-revoke-sub-bde2c551-gs8hc\" (UID: \"2126872e-0a4b-4d18-8985-d1aeb474052b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc" Oct 30 03:07:26.349339 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.349298 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2126872e-0a4b-4d18-8985-d1aeb474052b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bde2c551-gs8hc\" (UID: \"2126872e-0a4b-4d18-8985-d1aeb474052b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc" Oct 30 03:07:26.349339 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.349340 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2126872e-0a4b-4d18-8985-d1aeb474052b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bde2c551-gs8hc\" (UID: \"2126872e-0a4b-4d18-8985-d1aeb474052b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc" Oct 30 03:07:26.349625 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.349394 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl9ff\" (UniqueName: \"kubernetes.io/projected/2126872e-0a4b-4d18-8985-d1aeb474052b-kube-api-access-jl9ff\") pod \"maas-api-revoke-sub-bde2c551-gs8hc\" (UID: \"2126872e-0a4b-4d18-8985-d1aeb474052b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc" Oct 30 03:07:26.350130 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.350082 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2126872e-0a4b-4d18-8985-d1aeb474052b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bde2c551-gs8hc\" (UID: \"2126872e-0a4b-4d18-8985-d1aeb474052b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc" Oct 30 03:07:26.351692 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.351667 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2126872e-0a4b-4d18-8985-d1aeb474052b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bde2c551-gs8hc\" (UID: \"2126872e-0a4b-4d18-8985-d1aeb474052b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc" Oct 30 03:07:26.383474 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.383443 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl9ff\" (UniqueName: \"kubernetes.io/projected/2126872e-0a4b-4d18-8985-d1aeb474052b-kube-api-access-jl9ff\") pod \"maas-api-revoke-sub-bde2c551-gs8hc\" (UID: \"2126872e-0a4b-4d18-8985-d1aeb474052b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc" Oct 30 03:07:26.521202 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.521080 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc" Oct 30 03:07:26.824972 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.824851 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc"] Oct 30 03:07:26.829572 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:07:26.829536 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2126872e_0a4b_4d18_8985_d1aeb474052b.slice/crio-aa386441fc27dd19e8e266d2af721550d7f5cdd9c25e3a736ec3dc21dc22d96c WatchSource:0}: Error finding container aa386441fc27dd19e8e266d2af721550d7f5cdd9c25e3a736ec3dc21dc22d96c: Status 404 returned error can't find the container with id aa386441fc27dd19e8e266d2af721550d7f5cdd9c25e3a736ec3dc21dc22d96c Oct 30 03:07:26.893449 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:26.893413 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc" event={"ID":"2126872e-0a4b-4d18-8985-d1aeb474052b","Type":"ContainerStarted","Data":"aa386441fc27dd19e8e266d2af721550d7f5cdd9c25e3a736ec3dc21dc22d96c"} Oct 30 03:07:27.900233 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:27.900142 5998 generic.go:356] "Generic (PLEG): container finished" podID="2126872e-0a4b-4d18-8985-d1aeb474052b" containerID="222a2f9f8af24030daf3eb859755d4e8b078ba86495ca8dc32588cf47cbc1f90" exitCode=0 Oct 30 03:07:27.900233 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:27.900202 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc" event={"ID":"2126872e-0a4b-4d18-8985-d1aeb474052b","Type":"ContainerDied","Data":"222a2f9f8af24030daf3eb859755d4e8b078ba86495ca8dc32588cf47cbc1f90"} Oct 30 03:07:29.032132 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:29.032077 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc" Oct 30 03:07:29.074686 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:29.074650 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2126872e-0a4b-4d18-8985-d1aeb474052b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2126872e-0a4b-4d18-8985-d1aeb474052b-maas-api-xks-ca\") pod \"2126872e-0a4b-4d18-8985-d1aeb474052b\" (UID: \"2126872e-0a4b-4d18-8985-d1aeb474052b\") " Oct 30 03:07:29.074874 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:29.074782 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2126872e-0a4b-4d18-8985-d1aeb474052b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2126872e-0a4b-4d18-8985-d1aeb474052b-maas-api-openshift-ca\") pod \"2126872e-0a4b-4d18-8985-d1aeb474052b\" (UID: \"2126872e-0a4b-4d18-8985-d1aeb474052b\") " Oct 30 03:07:29.074874 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:29.074818 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2126872e-0a4b-4d18-8985-d1aeb474052b-kube-api-access-jl9ff\" (UniqueName: \"kubernetes.io/projected/2126872e-0a4b-4d18-8985-d1aeb474052b-kube-api-access-jl9ff\") pod \"2126872e-0a4b-4d18-8985-d1aeb474052b\" (UID: \"2126872e-0a4b-4d18-8985-d1aeb474052b\") " Oct 30 03:07:29.075214 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:29.075186 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2126872e-0a4b-4d18-8985-d1aeb474052b-maas-api-openshift-ca" pod "2126872e-0a4b-4d18-8985-d1aeb474052b" (UID: "2126872e-0a4b-4d18-8985-d1aeb474052b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:07:29.076909 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:29.076851 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2126872e-0a4b-4d18-8985-d1aeb474052b-maas-api-xks-ca" pod "2126872e-0a4b-4d18-8985-d1aeb474052b" (UID: "2126872e-0a4b-4d18-8985-d1aeb474052b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:07:29.077008 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:29.076905 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2126872e-0a4b-4d18-8985-d1aeb474052b-kube-api-access-jl9ff" pod "2126872e-0a4b-4d18-8985-d1aeb474052b" (UID: "2126872e-0a4b-4d18-8985-d1aeb474052b"). InnerVolumeSpecName "kube-api-access-jl9ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:07:29.176539 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:29.176504 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2126872e-0a4b-4d18-8985-d1aeb474052b-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:07:29.176539 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:29.176541 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl9ff\" (UniqueName: \"kubernetes.io/projected/2126872e-0a4b-4d18-8985-d1aeb474052b-kube-api-access-jl9ff\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:07:29.176757 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:29.176556 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2126872e-0a4b-4d18-8985-d1aeb474052b-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:07:29.911447 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:29.911412 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc" Oct 30 03:07:29.911630 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:29.911446 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bde2c551-gs8hc" event={"ID":"2126872e-0a4b-4d18-8985-d1aeb474052b","Type":"ContainerDied","Data":"aa386441fc27dd19e8e266d2af721550d7f5cdd9c25e3a736ec3dc21dc22d96c"} Oct 30 03:07:29.911630 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:29.911493 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa386441fc27dd19e8e266d2af721550d7f5cdd9c25e3a736ec3dc21dc22d96c" Oct 30 03:07:36.615526 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.615482 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8"] Oct 30 03:07:36.616176 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.616153 5998 cpu_manager.go:414] "containerMap: removing container" podUID="2126872e-0a4b-4d18-8985-d1aeb474052b" containerName="revoke-keys" Oct 30 03:07:36.616242 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.616179 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2126872e-0a4b-4d18-8985-d1aeb474052b" containerName="revoke-keys" Oct 30 03:07:36.616292 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.616262 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="2126872e-0a4b-4d18-8985-d1aeb474052b" containerName="revoke-keys" Oct 30 03:07:36.619959 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.619938 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8" Oct 30 03:07:36.637060 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.637029 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:07:36.637232 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.637100 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:07:36.649794 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.649759 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:07:36.649993 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.649877 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:07:36.651375 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.651353 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8"] Oct 30 03:07:36.749512 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.749473 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9338dd48-n5bf8\" (UID: \"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8" Oct 30 03:07:36.749703 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.749529 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9338dd48-n5bf8\" (UID: \"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8" Oct 30 03:07:36.749703 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.749617 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlcj2\" (UniqueName: \"kubernetes.io/projected/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-kube-api-access-tlcj2\") pod \"maas-api-revoke-sub-9338dd48-n5bf8\" (UID: \"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8" Oct 30 03:07:36.850888 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.850846 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9338dd48-n5bf8\" (UID: \"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8" Oct 30 03:07:36.851137 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.850919 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlcj2\" (UniqueName: \"kubernetes.io/projected/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-kube-api-access-tlcj2\") pod \"maas-api-revoke-sub-9338dd48-n5bf8\" (UID: \"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8" Oct 30 03:07:36.851137 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.850969 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9338dd48-n5bf8\" (UID: \"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8" Oct 30 03:07:36.851685 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.851665 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9338dd48-n5bf8\" (UID: \"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8" Oct 30 03:07:36.853377 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.853344 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9338dd48-n5bf8\" (UID: \"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8" Oct 30 03:07:36.874220 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.874142 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlcj2\" (UniqueName: \"kubernetes.io/projected/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-kube-api-access-tlcj2\") pod \"maas-api-revoke-sub-9338dd48-n5bf8\" (UID: \"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8" Oct 30 03:07:36.932956 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:36.932911 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8" Oct 30 03:07:37.239423 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:37.239383 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8"] Oct 30 03:07:37.242945 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:07:37.242912 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe8a28d_5d3e_447c_80f1_f3c9ef91cf69.slice/crio-c877b4afbf7cb09a2c79d2a163fcbbdda345126fe3b936b5c836ecf2f376c70a WatchSource:0}: Error finding container c877b4afbf7cb09a2c79d2a163fcbbdda345126fe3b936b5c836ecf2f376c70a: Status 404 returned error can't find the container with id c877b4afbf7cb09a2c79d2a163fcbbdda345126fe3b936b5c836ecf2f376c70a Oct 30 03:07:37.689286 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:37.689246 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:07:37.957141 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:37.957078 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8" event={"ID":"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69","Type":"ContainerStarted","Data":"6ad3892c82509e9c1ba6b8a15fa8d57e7e26fbc1ac288dea6e2e85bd18edce70"} Oct 30 03:07:37.957369 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:37.957154 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8" event={"ID":"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69","Type":"ContainerStarted","Data":"c877b4afbf7cb09a2c79d2a163fcbbdda345126fe3b936b5c836ecf2f376c70a"} Oct 30 03:07:37.997168 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:37.997112 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8" podStartSLOduration=1.99707837 podStartE2EDuration="1.99707837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:37.996554511 +0000 UTC m=+5887.575345592" watchObservedRunningTime="2026-10-30 03:07:37.99707837 +0000 UTC m=+5887.575869451" Oct 30 03:07:38.966937 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:38.966893 5998 generic.go:356] "Generic (PLEG): container finished" podID="2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69" containerID="6ad3892c82509e9c1ba6b8a15fa8d57e7e26fbc1ac288dea6e2e85bd18edce70" exitCode=0 Oct 30 03:07:38.967533 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:38.966970 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8" event={"ID":"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69","Type":"ContainerDied","Data":"6ad3892c82509e9c1ba6b8a15fa8d57e7e26fbc1ac288dea6e2e85bd18edce70"} Oct 30 03:07:40.496626 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:40.496523 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8" Oct 30 03:07:40.615113 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:40.605461 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-kube-api-access-tlcj2\" (UniqueName: \"kubernetes.io/projected/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-kube-api-access-tlcj2\") pod \"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69\" (UID: \"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69\") " Oct 30 03:07:40.615113 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:40.605533 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-maas-api-xks-ca\") pod \"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69\" (UID: \"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69\") " Oct 30 03:07:40.615113 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:40.605577 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-maas-api-openshift-ca\") pod \"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69\" (UID: \"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69\") " Oct 30 03:07:40.615113 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:40.613445 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-kube-api-access-tlcj2" pod "2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69" (UID: "2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69"). InnerVolumeSpecName "kube-api-access-tlcj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:07:40.616983 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:40.616238 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-maas-api-xks-ca" pod "2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69" (UID: "2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:07:40.616983 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:40.616595 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-maas-api-openshift-ca" pod "2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69" (UID: "2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:07:40.707045 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:40.706946 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlcj2\" (UniqueName: \"kubernetes.io/projected/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-kube-api-access-tlcj2\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:07:40.707045 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:40.706998 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:07:40.707045 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:40.707015 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:07:40.988991 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:40.988958 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8" Oct 30 03:07:40.989245 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:40.988987 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338dd48-n5bf8" event={"ID":"2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69","Type":"ContainerDied","Data":"c877b4afbf7cb09a2c79d2a163fcbbdda345126fe3b936b5c836ecf2f376c70a"} Oct 30 03:07:40.989245 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:40.989040 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c877b4afbf7cb09a2c79d2a163fcbbdda345126fe3b936b5c836ecf2f376c70a" Oct 30 03:07:44.461519 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:44.461482 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:48.051670 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:48.051631 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:07:53.393644 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:07:53.393596 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:07:55.217279 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:07:55.217235 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:08:01.479118 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:01.478172 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-28ssd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:08:01.479118 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:01.478260 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-28ssd" podUID="d8da8b22-5586-419c-9378-03d3d2d2c47f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:08:01.483270 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:01.483217 5998 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-b95w4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:08:01.483400 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:01.483302 5998 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-b95w4" podUID="0e843d95-ed87-4ea3-9ce8-d3f0103d9a80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:08:05.740438 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:05.740392 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:08:12.344677 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.344630 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46"] Oct 30 03:08:12.345213 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.345141 5998 cpu_manager.go:414] "containerMap: removing container" podUID="2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69" containerName="revoke-keys" Oct 30 03:08:12.345213 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.345163 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69" containerName="revoke-keys" Oct 30 03:08:12.345310 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.345247 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69" containerName="revoke-keys" Oct 30 03:08:12.348675 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.348657 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46" Oct 30 03:08:12.362745 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.362721 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:08:12.373966 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.373938 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:08:12.373966 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.373944 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:08:12.374171 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.373968 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:08:12.376289 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.376257 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46"] Oct 30 03:08:12.413004 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.412961 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc8bh\" (UniqueName: \"kubernetes.io/projected/8666bad4-75eb-4159-bd56-8a194a3ff6a8-kube-api-access-mc8bh\") pod \"maas-api-revoke-sub-5329dbbc-wrr46\" (UID: \"8666bad4-75eb-4159-bd56-8a194a3ff6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46" Oct 30 03:08:12.413222 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.413113 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8666bad4-75eb-4159-bd56-8a194a3ff6a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5329dbbc-wrr46\" (UID: \"8666bad4-75eb-4159-bd56-8a194a3ff6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46" Oct 30 03:08:12.413222 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.413187 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8666bad4-75eb-4159-bd56-8a194a3ff6a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5329dbbc-wrr46\" (UID: \"8666bad4-75eb-4159-bd56-8a194a3ff6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46" Oct 30 03:08:12.514504 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.514465 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8666bad4-75eb-4159-bd56-8a194a3ff6a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5329dbbc-wrr46\" (UID: \"8666bad4-75eb-4159-bd56-8a194a3ff6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46" Oct 30 03:08:12.514709 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.514513 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8666bad4-75eb-4159-bd56-8a194a3ff6a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5329dbbc-wrr46\" (UID: \"8666bad4-75eb-4159-bd56-8a194a3ff6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46" Oct 30 03:08:12.514709 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.514556 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc8bh\" (UniqueName: \"kubernetes.io/projected/8666bad4-75eb-4159-bd56-8a194a3ff6a8-kube-api-access-mc8bh\") pod \"maas-api-revoke-sub-5329dbbc-wrr46\" (UID: \"8666bad4-75eb-4159-bd56-8a194a3ff6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46" Oct 30 03:08:12.515286 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.515262 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8666bad4-75eb-4159-bd56-8a194a3ff6a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5329dbbc-wrr46\" (UID: \"8666bad4-75eb-4159-bd56-8a194a3ff6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46" Oct 30 03:08:12.516836 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.516817 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8666bad4-75eb-4159-bd56-8a194a3ff6a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5329dbbc-wrr46\" (UID: \"8666bad4-75eb-4159-bd56-8a194a3ff6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46" Oct 30 03:08:12.540296 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.540262 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc8bh\" (UniqueName: \"kubernetes.io/projected/8666bad4-75eb-4159-bd56-8a194a3ff6a8-kube-api-access-mc8bh\") pod \"maas-api-revoke-sub-5329dbbc-wrr46\" (UID: \"8666bad4-75eb-4159-bd56-8a194a3ff6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46" Oct 30 03:08:12.661264 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.661235 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46" Oct 30 03:08:12.959979 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:12.959883 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46"] Oct 30 03:08:12.964337 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:08:12.964308 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8666bad4_75eb_4159_bd56_8a194a3ff6a8.slice/crio-a0f0c48ca85fe1188ef4ff060c8e8a04e40f3dbceb0a39c197f41bed7d68a016 WatchSource:0}: Error finding container a0f0c48ca85fe1188ef4ff060c8e8a04e40f3dbceb0a39c197f41bed7d68a016: Status 404 returned error can't find the container with id a0f0c48ca85fe1188ef4ff060c8e8a04e40f3dbceb0a39c197f41bed7d68a016 Oct 30 03:08:13.285217 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:13.285131 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46" event={"ID":"8666bad4-75eb-4159-bd56-8a194a3ff6a8","Type":"ContainerStarted","Data":"f752dc94848b473eede9e2a33da479f8b16f382443c4bc3d29bcdbc537383729"} Oct 30 03:08:13.285217 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:13.285185 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46" event={"ID":"8666bad4-75eb-4159-bd56-8a194a3ff6a8","Type":"ContainerStarted","Data":"a0f0c48ca85fe1188ef4ff060c8e8a04e40f3dbceb0a39c197f41bed7d68a016"} Oct 30 03:08:13.326433 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:13.326368 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46" podStartSLOduration=1.32635511 podStartE2EDuration="1.32635511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:13.32424421 +0000 UTC m=+5922.903035290" watchObservedRunningTime="2026-10-30 03:08:13.32635511 +0000 UTC m=+5922.905146191" Oct 30 03:08:14.292499 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:14.292461 5998 generic.go:356] "Generic (PLEG): container finished" podID="8666bad4-75eb-4159-bd56-8a194a3ff6a8" containerID="f752dc94848b473eede9e2a33da479f8b16f382443c4bc3d29bcdbc537383729" exitCode=0 Oct 30 03:08:14.292920 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:14.292532 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46" event={"ID":"8666bad4-75eb-4159-bd56-8a194a3ff6a8","Type":"ContainerDied","Data":"f752dc94848b473eede9e2a33da479f8b16f382443c4bc3d29bcdbc537383729"} Oct 30 03:08:15.438311 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:15.438283 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46" Oct 30 03:08:15.543601 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:15.543545 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8666bad4-75eb-4159-bd56-8a194a3ff6a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8666bad4-75eb-4159-bd56-8a194a3ff6a8-maas-api-xks-ca\") pod \"8666bad4-75eb-4159-bd56-8a194a3ff6a8\" (UID: \"8666bad4-75eb-4159-bd56-8a194a3ff6a8\") " Oct 30 03:08:15.543822 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:15.543624 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8666bad4-75eb-4159-bd56-8a194a3ff6a8-kube-api-access-mc8bh\" (UniqueName: \"kubernetes.io/projected/8666bad4-75eb-4159-bd56-8a194a3ff6a8-kube-api-access-mc8bh\") pod \"8666bad4-75eb-4159-bd56-8a194a3ff6a8\" (UID: \"8666bad4-75eb-4159-bd56-8a194a3ff6a8\") " Oct 30 03:08:15.543822 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:15.543661 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8666bad4-75eb-4159-bd56-8a194a3ff6a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8666bad4-75eb-4159-bd56-8a194a3ff6a8-maas-api-openshift-ca\") pod \"8666bad4-75eb-4159-bd56-8a194a3ff6a8\" (UID: \"8666bad4-75eb-4159-bd56-8a194a3ff6a8\") " Oct 30 03:08:15.544049 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:15.544023 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8666bad4-75eb-4159-bd56-8a194a3ff6a8-maas-api-openshift-ca" pod "8666bad4-75eb-4159-bd56-8a194a3ff6a8" (UID: "8666bad4-75eb-4159-bd56-8a194a3ff6a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:08:15.545673 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:15.545643 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8666bad4-75eb-4159-bd56-8a194a3ff6a8-maas-api-xks-ca" pod "8666bad4-75eb-4159-bd56-8a194a3ff6a8" (UID: "8666bad4-75eb-4159-bd56-8a194a3ff6a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:08:15.545784 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:15.545695 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8666bad4-75eb-4159-bd56-8a194a3ff6a8-kube-api-access-mc8bh" pod "8666bad4-75eb-4159-bd56-8a194a3ff6a8" (UID: "8666bad4-75eb-4159-bd56-8a194a3ff6a8"). InnerVolumeSpecName "kube-api-access-mc8bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:08:15.645433 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:15.645393 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8666bad4-75eb-4159-bd56-8a194a3ff6a8-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:08:15.645433 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:15.645429 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc8bh\" (UniqueName: \"kubernetes.io/projected/8666bad4-75eb-4159-bd56-8a194a3ff6a8-kube-api-access-mc8bh\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:08:15.645433 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:15.645441 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8666bad4-75eb-4159-bd56-8a194a3ff6a8-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:08:16.304359 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:16.304331 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46" Oct 30 03:08:16.304542 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:16.304375 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5329dbbc-wrr46" event={"ID":"8666bad4-75eb-4159-bd56-8a194a3ff6a8","Type":"ContainerDied","Data":"a0f0c48ca85fe1188ef4ff060c8e8a04e40f3dbceb0a39c197f41bed7d68a016"} Oct 30 03:08:16.304542 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:16.304427 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0f0c48ca85fe1188ef4ff060c8e8a04e40f3dbceb0a39c197f41bed7d68a016" Oct 30 03:08:22.750537 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.750497 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7"] Oct 30 03:08:22.751198 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.751176 5998 cpu_manager.go:414] "containerMap: removing container" podUID="8666bad4-75eb-4159-bd56-8a194a3ff6a8" containerName="revoke-keys" Oct 30 03:08:22.751277 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.751203 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8666bad4-75eb-4159-bd56-8a194a3ff6a8" containerName="revoke-keys" Oct 30 03:08:22.751339 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.751324 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="8666bad4-75eb-4159-bd56-8a194a3ff6a8" containerName="revoke-keys" Oct 30 03:08:22.754329 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.754309 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7" Oct 30 03:08:22.768405 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.768378 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:08:22.780115 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.780067 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:08:22.780115 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.780069 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:08:22.780115 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.780069 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:08:22.782455 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.782427 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7"] Oct 30 03:08:22.809669 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.809630 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl58c\" (UniqueName: \"kubernetes.io/projected/dd243323-498d-41c8-a63a-05380de91423-kube-api-access-pl58c\") pod \"maas-api-revoke-sub-6f2d194d-kjpd7\" (UID: \"dd243323-498d-41c8-a63a-05380de91423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7" Oct 30 03:08:22.809842 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.809683 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd243323-498d-41c8-a63a-05380de91423-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f2d194d-kjpd7\" (UID: \"dd243323-498d-41c8-a63a-05380de91423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7" Oct 30 03:08:22.809888 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.809842 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd243323-498d-41c8-a63a-05380de91423-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f2d194d-kjpd7\" (UID: \"dd243323-498d-41c8-a63a-05380de91423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7" Oct 30 03:08:22.910676 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.910637 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd243323-498d-41c8-a63a-05380de91423-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f2d194d-kjpd7\" (UID: \"dd243323-498d-41c8-a63a-05380de91423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7" Oct 30 03:08:22.910676 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.910686 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl58c\" (UniqueName: \"kubernetes.io/projected/dd243323-498d-41c8-a63a-05380de91423-kube-api-access-pl58c\") pod \"maas-api-revoke-sub-6f2d194d-kjpd7\" (UID: \"dd243323-498d-41c8-a63a-05380de91423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7" Oct 30 03:08:22.910915 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.910722 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd243323-498d-41c8-a63a-05380de91423-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f2d194d-kjpd7\" (UID: \"dd243323-498d-41c8-a63a-05380de91423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7" Oct 30 03:08:22.911426 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.911402 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd243323-498d-41c8-a63a-05380de91423-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f2d194d-kjpd7\" (UID: \"dd243323-498d-41c8-a63a-05380de91423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7" Oct 30 03:08:22.913136 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.913108 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd243323-498d-41c8-a63a-05380de91423-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f2d194d-kjpd7\" (UID: \"dd243323-498d-41c8-a63a-05380de91423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7" Oct 30 03:08:22.935692 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:22.935661 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl58c\" (UniqueName: \"kubernetes.io/projected/dd243323-498d-41c8-a63a-05380de91423-kube-api-access-pl58c\") pod \"maas-api-revoke-sub-6f2d194d-kjpd7\" (UID: \"dd243323-498d-41c8-a63a-05380de91423\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7" Oct 30 03:08:23.067668 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:23.067568 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7" Oct 30 03:08:23.377798 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:23.377723 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7"] Oct 30 03:08:23.381645 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:08:23.381604 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd243323_498d_41c8_a63a_05380de91423.slice/crio-9f4bbbf94c8e48bb25f7dc4c9f6df53afca2147c0b174ab749288b55e9ad3e25 WatchSource:0}: Error finding container 9f4bbbf94c8e48bb25f7dc4c9f6df53afca2147c0b174ab749288b55e9ad3e25: Status 404 returned error can't find the container with id 9f4bbbf94c8e48bb25f7dc4c9f6df53afca2147c0b174ab749288b55e9ad3e25 Oct 30 03:08:23.876875 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:23.876838 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:08:24.350863 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:24.350799 5998 generic.go:356] "Generic (PLEG): container finished" podID="dd243323-498d-41c8-a63a-05380de91423" containerID="05dfe9574c17b670f53d199bb0de552fed5797ccc0b459f497dd3cd23a952cd8" exitCode=0 Oct 30 03:08:24.351074 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:24.351034 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7" event={"ID":"dd243323-498d-41c8-a63a-05380de91423","Type":"ContainerDied","Data":"05dfe9574c17b670f53d199bb0de552fed5797ccc0b459f497dd3cd23a952cd8"} Oct 30 03:08:24.351191 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:24.351125 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7" event={"ID":"dd243323-498d-41c8-a63a-05380de91423","Type":"ContainerStarted","Data":"9f4bbbf94c8e48bb25f7dc4c9f6df53afca2147c0b174ab749288b55e9ad3e25"} Oct 30 03:08:25.215856 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:08:25.215815 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:08:25.494265 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:25.494237 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7" Oct 30 03:08:25.526797 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:25.526757 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd243323-498d-41c8-a63a-05380de91423-kube-api-access-pl58c\" (UniqueName: \"kubernetes.io/projected/dd243323-498d-41c8-a63a-05380de91423-kube-api-access-pl58c\") pod \"dd243323-498d-41c8-a63a-05380de91423\" (UID: \"dd243323-498d-41c8-a63a-05380de91423\") " Oct 30 03:08:25.527053 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:25.526819 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd243323-498d-41c8-a63a-05380de91423-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd243323-498d-41c8-a63a-05380de91423-maas-api-xks-ca\") pod \"dd243323-498d-41c8-a63a-05380de91423\" (UID: \"dd243323-498d-41c8-a63a-05380de91423\") " Oct 30 03:08:25.527053 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:25.526853 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd243323-498d-41c8-a63a-05380de91423-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd243323-498d-41c8-a63a-05380de91423-maas-api-openshift-ca\") pod \"dd243323-498d-41c8-a63a-05380de91423\" (UID: \"dd243323-498d-41c8-a63a-05380de91423\") " Oct 30 03:08:25.527243 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:25.527218 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd243323-498d-41c8-a63a-05380de91423-maas-api-openshift-ca" pod "dd243323-498d-41c8-a63a-05380de91423" (UID: "dd243323-498d-41c8-a63a-05380de91423"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:08:25.528972 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:25.528947 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd243323-498d-41c8-a63a-05380de91423-maas-api-xks-ca" pod "dd243323-498d-41c8-a63a-05380de91423" (UID: "dd243323-498d-41c8-a63a-05380de91423"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:08:25.529084 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:25.529041 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd243323-498d-41c8-a63a-05380de91423-kube-api-access-pl58c" pod "dd243323-498d-41c8-a63a-05380de91423" (UID: "dd243323-498d-41c8-a63a-05380de91423"). InnerVolumeSpecName "kube-api-access-pl58c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:08:25.627918 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:25.627879 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl58c\" (UniqueName: \"kubernetes.io/projected/dd243323-498d-41c8-a63a-05380de91423-kube-api-access-pl58c\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:08:25.627918 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:25.627920 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd243323-498d-41c8-a63a-05380de91423-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:08:25.628184 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:25.627935 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd243323-498d-41c8-a63a-05380de91423-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:08:26.362803 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:26.362772 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7" Oct 30 03:08:26.362803 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:26.362782 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f2d194d-kjpd7" event={"ID":"dd243323-498d-41c8-a63a-05380de91423","Type":"ContainerDied","Data":"9f4bbbf94c8e48bb25f7dc4c9f6df53afca2147c0b174ab749288b55e9ad3e25"} Oct 30 03:08:26.363440 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:26.362828 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f4bbbf94c8e48bb25f7dc4c9f6df53afca2147c0b174ab749288b55e9ad3e25" Oct 30 03:08:33.084434 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.084395 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8"] Oct 30 03:08:33.085442 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.085419 5998 cpu_manager.go:414] "containerMap: removing container" podUID="dd243323-498d-41c8-a63a-05380de91423" containerName="revoke-keys" Oct 30 03:08:33.085512 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.085445 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd243323-498d-41c8-a63a-05380de91423" containerName="revoke-keys" Oct 30 03:08:33.085555 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.085528 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd243323-498d-41c8-a63a-05380de91423" containerName="revoke-keys" Oct 30 03:08:33.088626 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.088608 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8" Oct 30 03:08:33.102843 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.102811 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:08:33.114383 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.114340 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:08:33.114383 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.114339 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:08:33.114612 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.114397 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:08:33.116735 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.116709 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8"] Oct 30 03:08:33.120281 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.120234 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405d89d5-4a2c-4319-bffb-20e2bca0f82f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6da419fc-dqrx8\" (UID: \"405d89d5-4a2c-4319-bffb-20e2bca0f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8" Oct 30 03:08:33.120467 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.120388 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405d89d5-4a2c-4319-bffb-20e2bca0f82f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6da419fc-dqrx8\" (UID: \"405d89d5-4a2c-4319-bffb-20e2bca0f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8" Oct 30 03:08:33.120467 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.120455 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mkcp\" (UniqueName: \"kubernetes.io/projected/405d89d5-4a2c-4319-bffb-20e2bca0f82f-kube-api-access-2mkcp\") pod \"maas-api-revoke-sub-6da419fc-dqrx8\" (UID: \"405d89d5-4a2c-4319-bffb-20e2bca0f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8" Oct 30 03:08:33.221758 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.221713 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405d89d5-4a2c-4319-bffb-20e2bca0f82f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6da419fc-dqrx8\" (UID: \"405d89d5-4a2c-4319-bffb-20e2bca0f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8" Oct 30 03:08:33.221985 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.221800 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405d89d5-4a2c-4319-bffb-20e2bca0f82f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6da419fc-dqrx8\" (UID: \"405d89d5-4a2c-4319-bffb-20e2bca0f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8" Oct 30 03:08:33.221985 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.221838 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mkcp\" (UniqueName: \"kubernetes.io/projected/405d89d5-4a2c-4319-bffb-20e2bca0f82f-kube-api-access-2mkcp\") pod \"maas-api-revoke-sub-6da419fc-dqrx8\" (UID: \"405d89d5-4a2c-4319-bffb-20e2bca0f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8" Oct 30 03:08:33.222570 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.222544 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405d89d5-4a2c-4319-bffb-20e2bca0f82f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6da419fc-dqrx8\" (UID: \"405d89d5-4a2c-4319-bffb-20e2bca0f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8" Oct 30 03:08:33.224198 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.224172 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405d89d5-4a2c-4319-bffb-20e2bca0f82f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6da419fc-dqrx8\" (UID: \"405d89d5-4a2c-4319-bffb-20e2bca0f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8" Oct 30 03:08:33.244266 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.244233 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mkcp\" (UniqueName: \"kubernetes.io/projected/405d89d5-4a2c-4319-bffb-20e2bca0f82f-kube-api-access-2mkcp\") pod \"maas-api-revoke-sub-6da419fc-dqrx8\" (UID: \"405d89d5-4a2c-4319-bffb-20e2bca0f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8" Oct 30 03:08:33.401485 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.401399 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8" Oct 30 03:08:33.711515 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:33.711469 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8"] Oct 30 03:08:33.716256 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:08:33.716226 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405d89d5_4a2c_4319_bffb_20e2bca0f82f.slice/crio-7a51c63c165c532da9420d553a4390734f0a94bab0a1c463ebb9214daa698ecf WatchSource:0}: Error finding container 7a51c63c165c532da9420d553a4390734f0a94bab0a1c463ebb9214daa698ecf: Status 404 returned error can't find the container with id 7a51c63c165c532da9420d553a4390734f0a94bab0a1c463ebb9214daa698ecf Oct 30 03:08:34.420538 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:34.420498 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8" event={"ID":"405d89d5-4a2c-4319-bffb-20e2bca0f82f","Type":"ContainerStarted","Data":"8e0172a2f07dbfe6d49b6e372ea0484c822d05ef0940c9a75a923132ff3498d9"} Oct 30 03:08:34.420940 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:34.420561 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8" event={"ID":"405d89d5-4a2c-4319-bffb-20e2bca0f82f","Type":"ContainerStarted","Data":"7a51c63c165c532da9420d553a4390734f0a94bab0a1c463ebb9214daa698ecf"} Oct 30 03:08:34.460706 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:34.460655 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8" podStartSLOduration=1.46064237 podStartE2EDuration="1.46064237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:34.460432715 +0000 UTC m=+5944.039223798" watchObservedRunningTime="2026-10-30 03:08:34.46064237 +0000 UTC m=+5944.039433442" Oct 30 03:08:35.434399 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:35.434350 5998 generic.go:356] "Generic (PLEG): container finished" podID="405d89d5-4a2c-4319-bffb-20e2bca0f82f" containerID="8e0172a2f07dbfe6d49b6e372ea0484c822d05ef0940c9a75a923132ff3498d9" exitCode=0 Oct 30 03:08:35.434832 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:35.434430 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8" event={"ID":"405d89d5-4a2c-4319-bffb-20e2bca0f82f","Type":"ContainerDied","Data":"8e0172a2f07dbfe6d49b6e372ea0484c822d05ef0940c9a75a923132ff3498d9"} Oct 30 03:08:36.577311 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:36.577284 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8" Oct 30 03:08:36.656408 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:36.656370 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405d89d5-4a2c-4319-bffb-20e2bca0f82f-kube-api-access-2mkcp\" (UniqueName: \"kubernetes.io/projected/405d89d5-4a2c-4319-bffb-20e2bca0f82f-kube-api-access-2mkcp\") pod \"405d89d5-4a2c-4319-bffb-20e2bca0f82f\" (UID: \"405d89d5-4a2c-4319-bffb-20e2bca0f82f\") " Oct 30 03:08:36.656620 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:36.656453 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405d89d5-4a2c-4319-bffb-20e2bca0f82f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405d89d5-4a2c-4319-bffb-20e2bca0f82f-maas-api-openshift-ca\") pod \"405d89d5-4a2c-4319-bffb-20e2bca0f82f\" (UID: \"405d89d5-4a2c-4319-bffb-20e2bca0f82f\") " Oct 30 03:08:36.656620 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:36.656548 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/405d89d5-4a2c-4319-bffb-20e2bca0f82f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405d89d5-4a2c-4319-bffb-20e2bca0f82f-maas-api-xks-ca\") pod \"405d89d5-4a2c-4319-bffb-20e2bca0f82f\" (UID: \"405d89d5-4a2c-4319-bffb-20e2bca0f82f\") " Oct 30 03:08:36.656875 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:36.656849 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405d89d5-4a2c-4319-bffb-20e2bca0f82f-maas-api-openshift-ca" pod "405d89d5-4a2c-4319-bffb-20e2bca0f82f" (UID: "405d89d5-4a2c-4319-bffb-20e2bca0f82f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:08:36.658548 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:36.658525 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/405d89d5-4a2c-4319-bffb-20e2bca0f82f-maas-api-xks-ca" pod "405d89d5-4a2c-4319-bffb-20e2bca0f82f" (UID: "405d89d5-4a2c-4319-bffb-20e2bca0f82f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:08:36.658730 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:36.658706 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405d89d5-4a2c-4319-bffb-20e2bca0f82f-kube-api-access-2mkcp" pod "405d89d5-4a2c-4319-bffb-20e2bca0f82f" (UID: "405d89d5-4a2c-4319-bffb-20e2bca0f82f"). InnerVolumeSpecName "kube-api-access-2mkcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:08:36.757582 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:36.757489 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mkcp\" (UniqueName: \"kubernetes.io/projected/405d89d5-4a2c-4319-bffb-20e2bca0f82f-kube-api-access-2mkcp\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:08:36.757582 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:36.757524 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/405d89d5-4a2c-4319-bffb-20e2bca0f82f-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:08:36.757582 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:36.757535 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/405d89d5-4a2c-4319-bffb-20e2bca0f82f-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:08:37.446957 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:37.446922 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8" Oct 30 03:08:37.447157 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:37.446952 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6da419fc-dqrx8" event={"ID":"405d89d5-4a2c-4319-bffb-20e2bca0f82f","Type":"ContainerDied","Data":"7a51c63c165c532da9420d553a4390734f0a94bab0a1c463ebb9214daa698ecf"} Oct 30 03:08:37.447157 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:37.447005 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a51c63c165c532da9420d553a4390734f0a94bab0a1c463ebb9214daa698ecf" Oct 30 03:08:43.441278 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.441181 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg"] Oct 30 03:08:43.441789 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.441681 5998 cpu_manager.go:414] "containerMap: removing container" podUID="405d89d5-4a2c-4319-bffb-20e2bca0f82f" containerName="revoke-keys" Oct 30 03:08:43.441789 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.441699 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405d89d5-4a2c-4319-bffb-20e2bca0f82f" containerName="revoke-keys" Oct 30 03:08:43.441789 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.441777 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="405d89d5-4a2c-4319-bffb-20e2bca0f82f" containerName="revoke-keys" Oct 30 03:08:43.445160 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.445139 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg" Oct 30 03:08:43.459529 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.459503 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:08:43.459698 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.459595 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:08:43.471157 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.471124 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:08:43.471339 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.471168 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:08:43.474035 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.474008 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg"] Oct 30 03:08:43.524336 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.524299 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c10fe630-w2mvg\" (UID: \"e2e0e937-3a12-45c1-bbb6-ef31130c76ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg" Oct 30 03:08:43.524514 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.524359 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t9kl\" (UniqueName: \"kubernetes.io/projected/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-kube-api-access-8t9kl\") pod \"maas-api-revoke-sub-c10fe630-w2mvg\" (UID: \"e2e0e937-3a12-45c1-bbb6-ef31130c76ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg" Oct 30 03:08:43.524514 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.524449 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c10fe630-w2mvg\" (UID: \"e2e0e937-3a12-45c1-bbb6-ef31130c76ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg" Oct 30 03:08:43.625543 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.625496 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c10fe630-w2mvg\" (UID: \"e2e0e937-3a12-45c1-bbb6-ef31130c76ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg" Oct 30 03:08:43.625769 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.625583 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t9kl\" (UniqueName: \"kubernetes.io/projected/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-kube-api-access-8t9kl\") pod \"maas-api-revoke-sub-c10fe630-w2mvg\" (UID: \"e2e0e937-3a12-45c1-bbb6-ef31130c76ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg" Oct 30 03:08:43.625769 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.625617 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c10fe630-w2mvg\" (UID: \"e2e0e937-3a12-45c1-bbb6-ef31130c76ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg" Oct 30 03:08:43.626500 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.626466 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c10fe630-w2mvg\" (UID: \"e2e0e937-3a12-45c1-bbb6-ef31130c76ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg" Oct 30 03:08:43.628202 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.628175 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c10fe630-w2mvg\" (UID: \"e2e0e937-3a12-45c1-bbb6-ef31130c76ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg" Oct 30 03:08:43.648617 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.648591 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t9kl\" (UniqueName: \"kubernetes.io/projected/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-kube-api-access-8t9kl\") pod \"maas-api-revoke-sub-c10fe630-w2mvg\" (UID: \"e2e0e937-3a12-45c1-bbb6-ef31130c76ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg" Oct 30 03:08:43.758973 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:43.758882 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg" Oct 30 03:08:44.061164 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:44.061041 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg"] Oct 30 03:08:44.065665 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:08:44.065634 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e0e937_3a12_45c1_bbb6_ef31130c76ee.slice/crio-b0fe12cec0832d3d3fe52493227c13996588ddc86aa142fc764bc3a8eaa81b58 WatchSource:0}: Error finding container b0fe12cec0832d3d3fe52493227c13996588ddc86aa142fc764bc3a8eaa81b58: Status 404 returned error can't find the container with id b0fe12cec0832d3d3fe52493227c13996588ddc86aa142fc764bc3a8eaa81b58 Oct 30 03:08:44.488787 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:44.488741 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg" event={"ID":"e2e0e937-3a12-45c1-bbb6-ef31130c76ee","Type":"ContainerStarted","Data":"95a9368bfa4790b10bca28d67da6b72f87a144925523918d50d4b863738cae3a"} Oct 30 03:08:44.489247 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:44.488810 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg" event={"ID":"e2e0e937-3a12-45c1-bbb6-ef31130c76ee","Type":"ContainerStarted","Data":"b0fe12cec0832d3d3fe52493227c13996588ddc86aa142fc764bc3a8eaa81b58"} Oct 30 03:08:44.531082 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:44.530508 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg" podStartSLOduration=1.53049032 podStartE2EDuration="1.53049032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:44.52820674 +0000 UTC m=+5954.106997824" watchObservedRunningTime="2026-10-30 03:08:44.53049032 +0000 UTC m=+5954.109281402" Oct 30 03:08:44.591070 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:44.590988 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:45.268405 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:45.268302 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:45.498972 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:45.498933 5998 generic.go:356] "Generic (PLEG): container finished" podID="e2e0e937-3a12-45c1-bbb6-ef31130c76ee" containerID="95a9368bfa4790b10bca28d67da6b72f87a144925523918d50d4b863738cae3a" exitCode=0 Oct 30 03:08:45.499577 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:45.499036 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg" event={"ID":"e2e0e937-3a12-45c1-bbb6-ef31130c76ee","Type":"ContainerDied","Data":"95a9368bfa4790b10bca28d67da6b72f87a144925523918d50d4b863738cae3a"} Oct 30 03:08:46.998990 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:46.998962 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg" Oct 30 03:08:47.099793 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:47.099748 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-maas-api-openshift-ca\") pod \"e2e0e937-3a12-45c1-bbb6-ef31130c76ee\" (UID: \"e2e0e937-3a12-45c1-bbb6-ef31130c76ee\") " Oct 30 03:08:47.100021 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:47.099856 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-maas-api-xks-ca\") pod \"e2e0e937-3a12-45c1-bbb6-ef31130c76ee\" (UID: \"e2e0e937-3a12-45c1-bbb6-ef31130c76ee\") " Oct 30 03:08:47.100021 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:47.099949 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-kube-api-access-8t9kl\" (UniqueName: \"kubernetes.io/projected/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-kube-api-access-8t9kl\") pod \"e2e0e937-3a12-45c1-bbb6-ef31130c76ee\" (UID: \"e2e0e937-3a12-45c1-bbb6-ef31130c76ee\") " Oct 30 03:08:47.100228 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:47.100184 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-maas-api-openshift-ca" pod "e2e0e937-3a12-45c1-bbb6-ef31130c76ee" (UID: "e2e0e937-3a12-45c1-bbb6-ef31130c76ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:08:47.112113 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:47.100355 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:08:47.163116 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:47.149225 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-maas-api-xks-ca" pod "e2e0e937-3a12-45c1-bbb6-ef31130c76ee" (UID: "e2e0e937-3a12-45c1-bbb6-ef31130c76ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:08:47.163116 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:47.151301 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-kube-api-access-8t9kl" pod "e2e0e937-3a12-45c1-bbb6-ef31130c76ee" (UID: "e2e0e937-3a12-45c1-bbb6-ef31130c76ee"). InnerVolumeSpecName "kube-api-access-8t9kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:08:47.225114 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:47.224681 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t9kl\" (UniqueName: \"kubernetes.io/projected/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-kube-api-access-8t9kl\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:08:47.225114 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:47.224729 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e0e937-3a12-45c1-bbb6-ef31130c76ee-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:08:47.533378 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:47.533338 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg" event={"ID":"e2e0e937-3a12-45c1-bbb6-ef31130c76ee","Type":"ContainerDied","Data":"b0fe12cec0832d3d3fe52493227c13996588ddc86aa142fc764bc3a8eaa81b58"} Oct 30 03:08:47.533682 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:47.533661 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0fe12cec0832d3d3fe52493227c13996588ddc86aa142fc764bc3a8eaa81b58" Oct 30 03:08:47.533856 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:47.533843 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c10fe630-w2mvg" Oct 30 03:08:53.776059 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:53.776016 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb"] Oct 30 03:08:53.776679 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:53.776555 5998 cpu_manager.go:414] "containerMap: removing container" podUID="e2e0e937-3a12-45c1-bbb6-ef31130c76ee" containerName="revoke-keys" Oct 30 03:08:53.776679 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:53.776573 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e0e937-3a12-45c1-bbb6-ef31130c76ee" containerName="revoke-keys" Oct 30 03:08:53.776679 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:53.776648 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e0e937-3a12-45c1-bbb6-ef31130c76ee" containerName="revoke-keys" Oct 30 03:08:53.779313 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:53.779292 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb" Oct 30 03:08:53.793345 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:53.793315 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:08:53.804677 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:53.804645 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:08:53.804881 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:53.804743 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:08:53.805624 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:53.805336 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:08:53.807911 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:53.807884 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb"] Oct 30 03:08:53.897835 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:53.897801 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e56e09aa-2161-4528-9238-773299002451-maas-api-xks-ca\") pod \"maas-api-revoke-sub-485ab6cf-kv7pb\" (UID: \"e56e09aa-2161-4528-9238-773299002451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb" Oct 30 03:08:53.898011 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:53.897852 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf44m\" (UniqueName: \"kubernetes.io/projected/e56e09aa-2161-4528-9238-773299002451-kube-api-access-zf44m\") pod \"maas-api-revoke-sub-485ab6cf-kv7pb\" (UID: \"e56e09aa-2161-4528-9238-773299002451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb" Oct 30 03:08:53.898011 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:53.897969 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e56e09aa-2161-4528-9238-773299002451-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-485ab6cf-kv7pb\" (UID: \"e56e09aa-2161-4528-9238-773299002451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb" Oct 30 03:08:53.998482 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:53.998444 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e56e09aa-2161-4528-9238-773299002451-maas-api-xks-ca\") pod \"maas-api-revoke-sub-485ab6cf-kv7pb\" (UID: \"e56e09aa-2161-4528-9238-773299002451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb" Oct 30 03:08:53.998665 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:53.998491 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf44m\" (UniqueName: \"kubernetes.io/projected/e56e09aa-2161-4528-9238-773299002451-kube-api-access-zf44m\") pod \"maas-api-revoke-sub-485ab6cf-kv7pb\" (UID: \"e56e09aa-2161-4528-9238-773299002451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb" Oct 30 03:08:53.998665 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:53.998539 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e56e09aa-2161-4528-9238-773299002451-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-485ab6cf-kv7pb\" (UID: \"e56e09aa-2161-4528-9238-773299002451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb" Oct 30 03:08:53.999386 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:53.999355 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e56e09aa-2161-4528-9238-773299002451-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-485ab6cf-kv7pb\" (UID: \"e56e09aa-2161-4528-9238-773299002451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb" Oct 30 03:08:54.000969 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:54.000932 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e56e09aa-2161-4528-9238-773299002451-maas-api-xks-ca\") pod \"maas-api-revoke-sub-485ab6cf-kv7pb\" (UID: \"e56e09aa-2161-4528-9238-773299002451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb" Oct 30 03:08:54.021217 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:54.021181 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf44m\" (UniqueName: \"kubernetes.io/projected/e56e09aa-2161-4528-9238-773299002451-kube-api-access-zf44m\") pod \"maas-api-revoke-sub-485ab6cf-kv7pb\" (UID: \"e56e09aa-2161-4528-9238-773299002451\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb" Oct 30 03:08:54.093051 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:54.092962 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb" Oct 30 03:08:54.396781 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:54.396695 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb"] Oct 30 03:08:54.401723 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:08:54.401694 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode56e09aa_2161_4528_9238_773299002451.slice/crio-15b3c68930391095deb5e470fccd56df9cd07d39bb68d346a0d5792d80e538e6 WatchSource:0}: Error finding container 15b3c68930391095deb5e470fccd56df9cd07d39bb68d346a0d5792d80e538e6: Status 404 returned error can't find the container with id 15b3c68930391095deb5e470fccd56df9cd07d39bb68d346a0d5792d80e538e6 Oct 30 03:08:54.571872 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:54.571834 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb" event={"ID":"e56e09aa-2161-4528-9238-773299002451","Type":"ContainerStarted","Data":"15b3c68930391095deb5e470fccd56df9cd07d39bb68d346a0d5792d80e538e6"} Oct 30 03:08:55.219379 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:08:55.219340 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:08:55.583006 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:55.582901 5998 generic.go:356] "Generic (PLEG): container finished" podID="e56e09aa-2161-4528-9238-773299002451" containerID="9103f780d2b1e3ebe109bea50e52313196be9c08653f0c34e0b1c4fccde97453" exitCode=0 Oct 30 03:08:55.583006 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:55.582964 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb" event={"ID":"e56e09aa-2161-4528-9238-773299002451","Type":"ContainerDied","Data":"9103f780d2b1e3ebe109bea50e52313196be9c08653f0c34e0b1c4fccde97453"} Oct 30 03:08:56.490003 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:56.489968 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:57.075752 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:57.075331 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb" Oct 30 03:08:57.185185 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:57.176423 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e56e09aa-2161-4528-9238-773299002451-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e56e09aa-2161-4528-9238-773299002451-maas-api-openshift-ca\") pod \"e56e09aa-2161-4528-9238-773299002451\" (UID: \"e56e09aa-2161-4528-9238-773299002451\") " Oct 30 03:08:57.185185 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:57.176564 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e56e09aa-2161-4528-9238-773299002451-kube-api-access-zf44m\" (UniqueName: \"kubernetes.io/projected/e56e09aa-2161-4528-9238-773299002451-kube-api-access-zf44m\") pod \"e56e09aa-2161-4528-9238-773299002451\" (UID: \"e56e09aa-2161-4528-9238-773299002451\") " Oct 30 03:08:57.185185 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:57.176640 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e56e09aa-2161-4528-9238-773299002451-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e56e09aa-2161-4528-9238-773299002451-maas-api-xks-ca\") pod \"e56e09aa-2161-4528-9238-773299002451\" (UID: \"e56e09aa-2161-4528-9238-773299002451\") " Oct 30 03:08:57.185185 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:57.183293 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e56e09aa-2161-4528-9238-773299002451-maas-api-xks-ca" pod "e56e09aa-2161-4528-9238-773299002451" (UID: "e56e09aa-2161-4528-9238-773299002451"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:08:57.185185 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:57.183751 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e56e09aa-2161-4528-9238-773299002451-maas-api-openshift-ca" pod "e56e09aa-2161-4528-9238-773299002451" (UID: "e56e09aa-2161-4528-9238-773299002451"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:08:57.187131 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:57.186545 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e56e09aa-2161-4528-9238-773299002451-kube-api-access-zf44m" pod "e56e09aa-2161-4528-9238-773299002451" (UID: "e56e09aa-2161-4528-9238-773299002451"). InnerVolumeSpecName "kube-api-access-zf44m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:08:57.278538 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:57.278203 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf44m\" (UniqueName: \"kubernetes.io/projected/e56e09aa-2161-4528-9238-773299002451-kube-api-access-zf44m\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:08:57.278538 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:57.278245 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e56e09aa-2161-4528-9238-773299002451-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:08:57.278538 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:57.278262 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e56e09aa-2161-4528-9238-773299002451-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:08:57.613112 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:57.609826 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb" event={"ID":"e56e09aa-2161-4528-9238-773299002451","Type":"ContainerDied","Data":"15b3c68930391095deb5e470fccd56df9cd07d39bb68d346a0d5792d80e538e6"} Oct 30 03:08:57.613112 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:57.609894 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15b3c68930391095deb5e470fccd56df9cd07d39bb68d346a0d5792d80e538e6" Oct 30 03:08:57.613112 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:08:57.609970 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-485ab6cf-kv7pb" Oct 30 03:09:04.101606 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.101561 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k"] Oct 30 03:09:04.102056 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.102040 5998 cpu_manager.go:414] "containerMap: removing container" podUID="e56e09aa-2161-4528-9238-773299002451" containerName="revoke-keys" Oct 30 03:09:04.102142 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.102056 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e56e09aa-2161-4528-9238-773299002451" containerName="revoke-keys" Oct 30 03:09:04.102183 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.102148 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="e56e09aa-2161-4528-9238-773299002451" containerName="revoke-keys" Oct 30 03:09:04.105282 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.105258 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k" Oct 30 03:09:04.131139 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.131098 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:09:04.131348 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.131170 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:09:04.131348 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.131184 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:09:04.133236 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.133207 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:09:04.135348 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.135312 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k"] Oct 30 03:09:04.140788 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.140760 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70785ea-4edd-466b-99e9-dba594675a9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d4ef071-r5z4k\" (UID: \"d70785ea-4edd-466b-99e9-dba594675a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k" Oct 30 03:09:04.140934 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.140805 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sr27\" (UniqueName: \"kubernetes.io/projected/d70785ea-4edd-466b-99e9-dba594675a9e-kube-api-access-4sr27\") pod \"maas-api-revoke-sub-9d4ef071-r5z4k\" (UID: \"d70785ea-4edd-466b-99e9-dba594675a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k" Oct 30 03:09:04.140934 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.140832 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70785ea-4edd-466b-99e9-dba594675a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d4ef071-r5z4k\" (UID: \"d70785ea-4edd-466b-99e9-dba594675a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k" Oct 30 03:09:04.241429 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.241386 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sr27\" (UniqueName: \"kubernetes.io/projected/d70785ea-4edd-466b-99e9-dba594675a9e-kube-api-access-4sr27\") pod \"maas-api-revoke-sub-9d4ef071-r5z4k\" (UID: \"d70785ea-4edd-466b-99e9-dba594675a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k" Oct 30 03:09:04.241616 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.241437 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70785ea-4edd-466b-99e9-dba594675a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d4ef071-r5z4k\" (UID: \"d70785ea-4edd-466b-99e9-dba594675a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k" Oct 30 03:09:04.241616 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.241533 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70785ea-4edd-466b-99e9-dba594675a9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d4ef071-r5z4k\" (UID: \"d70785ea-4edd-466b-99e9-dba594675a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k" Oct 30 03:09:04.242336 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.242312 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70785ea-4edd-466b-99e9-dba594675a9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d4ef071-r5z4k\" (UID: \"d70785ea-4edd-466b-99e9-dba594675a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k" Oct 30 03:09:04.243972 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.243952 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70785ea-4edd-466b-99e9-dba594675a9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d4ef071-r5z4k\" (UID: \"d70785ea-4edd-466b-99e9-dba594675a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k" Oct 30 03:09:04.263996 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.263965 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sr27\" (UniqueName: \"kubernetes.io/projected/d70785ea-4edd-466b-99e9-dba594675a9e-kube-api-access-4sr27\") pod \"maas-api-revoke-sub-9d4ef071-r5z4k\" (UID: \"d70785ea-4edd-466b-99e9-dba594675a9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k" Oct 30 03:09:04.418398 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.418349 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k" Oct 30 03:09:04.720810 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:04.720727 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k"] Oct 30 03:09:04.725473 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:09:04.725440 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70785ea_4edd_466b_99e9_dba594675a9e.slice/crio-a9394a2a47c29147349ac238e92d64650eb0221dbea7c50c2bad51465212f421 WatchSource:0}: Error finding container a9394a2a47c29147349ac238e92d64650eb0221dbea7c50c2bad51465212f421: Status 404 returned error can't find the container with id a9394a2a47c29147349ac238e92d64650eb0221dbea7c50c2bad51465212f421 Oct 30 03:09:05.667224 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:05.667189 5998 generic.go:356] "Generic (PLEG): container finished" podID="d70785ea-4edd-466b-99e9-dba594675a9e" containerID="c4525d7aaf601957d77ff3bbabb8bc73333e13c2d0ec130e2940195130c6c50e" exitCode=0 Oct 30 03:09:05.667617 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:05.667252 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k" event={"ID":"d70785ea-4edd-466b-99e9-dba594675a9e","Type":"ContainerDied","Data":"c4525d7aaf601957d77ff3bbabb8bc73333e13c2d0ec130e2940195130c6c50e"} Oct 30 03:09:05.667617 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:05.667298 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k" event={"ID":"d70785ea-4edd-466b-99e9-dba594675a9e","Type":"ContainerStarted","Data":"a9394a2a47c29147349ac238e92d64650eb0221dbea7c50c2bad51465212f421"} Oct 30 03:09:07.200117 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:07.199271 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k" Oct 30 03:09:07.403565 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:07.400017 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70785ea-4edd-466b-99e9-dba594675a9e-kube-api-access-4sr27\" (UniqueName: \"kubernetes.io/projected/d70785ea-4edd-466b-99e9-dba594675a9e-kube-api-access-4sr27\") pod \"d70785ea-4edd-466b-99e9-dba594675a9e\" (UID: \"d70785ea-4edd-466b-99e9-dba594675a9e\") " Oct 30 03:09:07.403565 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:07.400221 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d70785ea-4edd-466b-99e9-dba594675a9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70785ea-4edd-466b-99e9-dba594675a9e-maas-api-xks-ca\") pod \"d70785ea-4edd-466b-99e9-dba594675a9e\" (UID: \"d70785ea-4edd-466b-99e9-dba594675a9e\") " Oct 30 03:09:07.403565 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:07.400291 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d70785ea-4edd-466b-99e9-dba594675a9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70785ea-4edd-466b-99e9-dba594675a9e-maas-api-openshift-ca\") pod \"d70785ea-4edd-466b-99e9-dba594675a9e\" (UID: \"d70785ea-4edd-466b-99e9-dba594675a9e\") " Oct 30 03:09:07.404534 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:07.404070 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70785ea-4edd-466b-99e9-dba594675a9e-maas-api-openshift-ca" pod "d70785ea-4edd-466b-99e9-dba594675a9e" (UID: "d70785ea-4edd-466b-99e9-dba594675a9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:09:07.446410 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:07.446355 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d70785ea-4edd-466b-99e9-dba594675a9e-maas-api-xks-ca" pod "d70785ea-4edd-466b-99e9-dba594675a9e" (UID: "d70785ea-4edd-466b-99e9-dba594675a9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:09:07.452318 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:07.452260 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70785ea-4edd-466b-99e9-dba594675a9e-kube-api-access-4sr27" pod "d70785ea-4edd-466b-99e9-dba594675a9e" (UID: "d70785ea-4edd-466b-99e9-dba594675a9e"). InnerVolumeSpecName "kube-api-access-4sr27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:09:07.503662 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:07.503563 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70785ea-4edd-466b-99e9-dba594675a9e-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:09:07.503662 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:07.503615 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70785ea-4edd-466b-99e9-dba594675a9e-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:09:07.503662 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:07.503634 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sr27\" (UniqueName: \"kubernetes.io/projected/d70785ea-4edd-466b-99e9-dba594675a9e-kube-api-access-4sr27\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:09:07.686166 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:07.686135 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k" Oct 30 03:09:07.686166 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:07.686147 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4ef071-r5z4k" event={"ID":"d70785ea-4edd-466b-99e9-dba594675a9e","Type":"ContainerDied","Data":"a9394a2a47c29147349ac238e92d64650eb0221dbea7c50c2bad51465212f421"} Oct 30 03:09:07.686409 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:07.686192 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9394a2a47c29147349ac238e92d64650eb0221dbea7c50c2bad51465212f421" Oct 30 03:09:14.433792 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.433755 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl"] Oct 30 03:09:14.434455 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.434431 5998 cpu_manager.go:414] "containerMap: removing container" podUID="d70785ea-4edd-466b-99e9-dba594675a9e" containerName="revoke-keys" Oct 30 03:09:14.434561 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.434456 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70785ea-4edd-466b-99e9-dba594675a9e" containerName="revoke-keys" Oct 30 03:09:14.434626 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.434566 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70785ea-4edd-466b-99e9-dba594675a9e" containerName="revoke-keys" Oct 30 03:09:14.437936 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.437912 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl" Oct 30 03:09:14.451954 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.451925 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:09:14.452130 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.452024 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:09:14.452130 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.452047 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:09:14.462122 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.462051 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825aabe5-05da-4610-a8e4-c312ba9d1327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-590d399f-2kwtl\" (UID: \"825aabe5-05da-4610-a8e4-c312ba9d1327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl" Oct 30 03:09:14.462287 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.462124 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8hl7\" (UniqueName: \"kubernetes.io/projected/825aabe5-05da-4610-a8e4-c312ba9d1327-kube-api-access-g8hl7\") pod \"maas-api-revoke-sub-590d399f-2kwtl\" (UID: \"825aabe5-05da-4610-a8e4-c312ba9d1327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl" Oct 30 03:09:14.462287 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.462160 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825aabe5-05da-4610-a8e4-c312ba9d1327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-590d399f-2kwtl\" (UID: \"825aabe5-05da-4610-a8e4-c312ba9d1327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl" Oct 30 03:09:14.465995 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.465511 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:09:14.466764 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.466730 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl"] Oct 30 03:09:14.563595 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.563555 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825aabe5-05da-4610-a8e4-c312ba9d1327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-590d399f-2kwtl\" (UID: \"825aabe5-05da-4610-a8e4-c312ba9d1327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl" Oct 30 03:09:14.563923 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.563598 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8hl7\" (UniqueName: \"kubernetes.io/projected/825aabe5-05da-4610-a8e4-c312ba9d1327-kube-api-access-g8hl7\") pod \"maas-api-revoke-sub-590d399f-2kwtl\" (UID: \"825aabe5-05da-4610-a8e4-c312ba9d1327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl" Oct 30 03:09:14.563923 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.563635 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825aabe5-05da-4610-a8e4-c312ba9d1327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-590d399f-2kwtl\" (UID: \"825aabe5-05da-4610-a8e4-c312ba9d1327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl" Oct 30 03:09:14.564381 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.564358 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825aabe5-05da-4610-a8e4-c312ba9d1327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-590d399f-2kwtl\" (UID: \"825aabe5-05da-4610-a8e4-c312ba9d1327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl" Oct 30 03:09:14.565988 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.565970 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825aabe5-05da-4610-a8e4-c312ba9d1327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-590d399f-2kwtl\" (UID: \"825aabe5-05da-4610-a8e4-c312ba9d1327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl" Oct 30 03:09:14.586041 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.586005 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8hl7\" (UniqueName: \"kubernetes.io/projected/825aabe5-05da-4610-a8e4-c312ba9d1327-kube-api-access-g8hl7\") pod \"maas-api-revoke-sub-590d399f-2kwtl\" (UID: \"825aabe5-05da-4610-a8e4-c312ba9d1327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl" Oct 30 03:09:14.751708 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:14.751623 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl" Oct 30 03:09:15.051952 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:15.051863 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl"] Oct 30 03:09:15.055973 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:09:15.055925 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825aabe5_05da_4610_a8e4_c312ba9d1327.slice/crio-35e5d0e89e2aed7756a3e7a931d972a31e4de9172a2577215c4e7ad100855cbc WatchSource:0}: Error finding container 35e5d0e89e2aed7756a3e7a931d972a31e4de9172a2577215c4e7ad100855cbc: Status 404 returned error can't find the container with id 35e5d0e89e2aed7756a3e7a931d972a31e4de9172a2577215c4e7ad100855cbc Oct 30 03:09:15.734582 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:15.734543 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl" event={"ID":"825aabe5-05da-4610-a8e4-c312ba9d1327","Type":"ContainerStarted","Data":"67299a03763d796f5ae125f13b1130883a76476dfa14ac5eb1074069d3b51fa5"} Oct 30 03:09:15.735084 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:15.734600 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl" event={"ID":"825aabe5-05da-4610-a8e4-c312ba9d1327","Type":"ContainerStarted","Data":"35e5d0e89e2aed7756a3e7a931d972a31e4de9172a2577215c4e7ad100855cbc"} Oct 30 03:09:15.775031 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:15.774973 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl" podStartSLOduration=1.7749608110000001 podStartE2EDuration="1.774960811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:15.773379067 +0000 UTC m=+5985.352170141" watchObservedRunningTime="2026-10-30 03:09:15.774960811 +0000 UTC m=+5985.353751910" Oct 30 03:09:16.745797 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:16.745751 5998 generic.go:356] "Generic (PLEG): container finished" podID="825aabe5-05da-4610-a8e4-c312ba9d1327" containerID="67299a03763d796f5ae125f13b1130883a76476dfa14ac5eb1074069d3b51fa5" exitCode=0 Oct 30 03:09:16.746370 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:16.745870 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl" event={"ID":"825aabe5-05da-4610-a8e4-c312ba9d1327","Type":"ContainerDied","Data":"67299a03763d796f5ae125f13b1130883a76476dfa14ac5eb1074069d3b51fa5"} Oct 30 03:09:18.329607 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:18.329267 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl" Oct 30 03:09:18.377022 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:18.376547 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/825aabe5-05da-4610-a8e4-c312ba9d1327-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825aabe5-05da-4610-a8e4-c312ba9d1327-maas-api-xks-ca\") pod \"825aabe5-05da-4610-a8e4-c312ba9d1327\" (UID: \"825aabe5-05da-4610-a8e4-c312ba9d1327\") " Oct 30 03:09:18.377375 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:18.377349 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/825aabe5-05da-4610-a8e4-c312ba9d1327-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825aabe5-05da-4610-a8e4-c312ba9d1327-maas-api-openshift-ca\") pod \"825aabe5-05da-4610-a8e4-c312ba9d1327\" (UID: \"825aabe5-05da-4610-a8e4-c312ba9d1327\") " Oct 30 03:09:18.377690 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:18.377669 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/825aabe5-05da-4610-a8e4-c312ba9d1327-kube-api-access-g8hl7\" (UniqueName: \"kubernetes.io/projected/825aabe5-05da-4610-a8e4-c312ba9d1327-kube-api-access-g8hl7\") pod \"825aabe5-05da-4610-a8e4-c312ba9d1327\" (UID: \"825aabe5-05da-4610-a8e4-c312ba9d1327\") " Oct 30 03:09:18.378164 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:18.378118 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825aabe5-05da-4610-a8e4-c312ba9d1327-maas-api-openshift-ca" pod "825aabe5-05da-4610-a8e4-c312ba9d1327" (UID: "825aabe5-05da-4610-a8e4-c312ba9d1327"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:09:18.379834 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:18.379806 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/825aabe5-05da-4610-a8e4-c312ba9d1327-maas-api-xks-ca" pod "825aabe5-05da-4610-a8e4-c312ba9d1327" (UID: "825aabe5-05da-4610-a8e4-c312ba9d1327"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:09:18.380403 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:18.380377 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825aabe5-05da-4610-a8e4-c312ba9d1327-kube-api-access-g8hl7" pod "825aabe5-05da-4610-a8e4-c312ba9d1327" (UID: "825aabe5-05da-4610-a8e4-c312ba9d1327"). InnerVolumeSpecName "kube-api-access-g8hl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:09:18.478986 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:18.478869 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/825aabe5-05da-4610-a8e4-c312ba9d1327-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:09:18.478986 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:18.478933 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/825aabe5-05da-4610-a8e4-c312ba9d1327-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:09:18.478986 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:18.478950 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8hl7\" (UniqueName: \"kubernetes.io/projected/825aabe5-05da-4610-a8e4-c312ba9d1327-kube-api-access-g8hl7\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:09:18.794934 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:18.794899 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl" Oct 30 03:09:18.795136 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:18.794905 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-590d399f-2kwtl" event={"ID":"825aabe5-05da-4610-a8e4-c312ba9d1327","Type":"ContainerDied","Data":"35e5d0e89e2aed7756a3e7a931d972a31e4de9172a2577215c4e7ad100855cbc"} Oct 30 03:09:18.795136 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:18.795065 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35e5d0e89e2aed7756a3e7a931d972a31e4de9172a2577215c4e7ad100855cbc" Oct 30 03:09:24.767022 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.766978 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn"] Oct 30 03:09:24.767577 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.767561 5998 cpu_manager.go:414] "containerMap: removing container" podUID="825aabe5-05da-4610-a8e4-c312ba9d1327" containerName="revoke-keys" Oct 30 03:09:24.767625 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.767582 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="825aabe5-05da-4610-a8e4-c312ba9d1327" containerName="revoke-keys" Oct 30 03:09:24.767667 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.767660 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="825aabe5-05da-4610-a8e4-c312ba9d1327" containerName="revoke-keys" Oct 30 03:09:24.771067 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.771046 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn" Oct 30 03:09:24.785115 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.785071 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:09:24.796710 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.796685 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:09:24.796892 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.796754 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:09:24.796892 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.796762 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:09:24.799692 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.799666 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn"] Oct 30 03:09:24.839348 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.839304 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed2b192-3c98-4542-a416-ce77621acf8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4260dc98-cgkzn\" (UID: \"bed2b192-3c98-4542-a416-ce77621acf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn" Oct 30 03:09:24.839540 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.839413 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed2b192-3c98-4542-a416-ce77621acf8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4260dc98-cgkzn\" (UID: \"bed2b192-3c98-4542-a416-ce77621acf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn" Oct 30 03:09:24.839540 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.839470 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dfks\" (UniqueName: \"kubernetes.io/projected/bed2b192-3c98-4542-a416-ce77621acf8d-kube-api-access-8dfks\") pod \"maas-api-revoke-sub-4260dc98-cgkzn\" (UID: \"bed2b192-3c98-4542-a416-ce77621acf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn" Oct 30 03:09:24.940370 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.940326 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed2b192-3c98-4542-a416-ce77621acf8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4260dc98-cgkzn\" (UID: \"bed2b192-3c98-4542-a416-ce77621acf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn" Oct 30 03:09:24.940595 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.940381 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed2b192-3c98-4542-a416-ce77621acf8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4260dc98-cgkzn\" (UID: \"bed2b192-3c98-4542-a416-ce77621acf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn" Oct 30 03:09:24.940595 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.940413 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dfks\" (UniqueName: \"kubernetes.io/projected/bed2b192-3c98-4542-a416-ce77621acf8d-kube-api-access-8dfks\") pod \"maas-api-revoke-sub-4260dc98-cgkzn\" (UID: \"bed2b192-3c98-4542-a416-ce77621acf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn" Oct 30 03:09:24.941120 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.941059 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed2b192-3c98-4542-a416-ce77621acf8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4260dc98-cgkzn\" (UID: \"bed2b192-3c98-4542-a416-ce77621acf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn" Oct 30 03:09:24.942804 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.942782 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed2b192-3c98-4542-a416-ce77621acf8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4260dc98-cgkzn\" (UID: \"bed2b192-3c98-4542-a416-ce77621acf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn" Oct 30 03:09:24.963150 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:24.963112 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dfks\" (UniqueName: \"kubernetes.io/projected/bed2b192-3c98-4542-a416-ce77621acf8d-kube-api-access-8dfks\") pod \"maas-api-revoke-sub-4260dc98-cgkzn\" (UID: \"bed2b192-3c98-4542-a416-ce77621acf8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn" Oct 30 03:09:25.083900 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:25.083808 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn" Oct 30 03:09:25.218000 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:09:25.217958 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:09:25.387407 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:25.387327 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn"] Oct 30 03:09:25.392317 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:09:25.392277 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed2b192_3c98_4542_a416_ce77621acf8d.slice/crio-3db3b8852b2d1d5c6925e9579ad42ec7efe5b6c29de0496b8c80bcefef29dd7d WatchSource:0}: Error finding container 3db3b8852b2d1d5c6925e9579ad42ec7efe5b6c29de0496b8c80bcefef29dd7d: Status 404 returned error can't find the container with id 3db3b8852b2d1d5c6925e9579ad42ec7efe5b6c29de0496b8c80bcefef29dd7d Oct 30 03:09:25.838796 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:25.838752 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn" event={"ID":"bed2b192-3c98-4542-a416-ce77621acf8d","Type":"ContainerStarted","Data":"34ff0bf8d0f4b444001ea3dbc0493daf7067b370291e042f488d060611cde2a2"} Oct 30 03:09:25.839355 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:25.838824 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn" event={"ID":"bed2b192-3c98-4542-a416-ce77621acf8d","Type":"ContainerStarted","Data":"3db3b8852b2d1d5c6925e9579ad42ec7efe5b6c29de0496b8c80bcefef29dd7d"} Oct 30 03:09:25.879984 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:25.879924 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn" podStartSLOduration=1.879907277 podStartE2EDuration="1.879907277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:25.878157489 +0000 UTC m=+5995.456948570" watchObservedRunningTime="2026-10-30 03:09:25.879907277 +0000 UTC m=+5995.458698357" Oct 30 03:09:26.850369 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:26.850259 5998 generic.go:356] "Generic (PLEG): container finished" podID="bed2b192-3c98-4542-a416-ce77621acf8d" containerID="34ff0bf8d0f4b444001ea3dbc0493daf7067b370291e042f488d060611cde2a2" exitCode=0 Oct 30 03:09:26.850879 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:26.850366 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn" event={"ID":"bed2b192-3c98-4542-a416-ce77621acf8d","Type":"ContainerDied","Data":"34ff0bf8d0f4b444001ea3dbc0493daf7067b370291e042f488d060611cde2a2"} Oct 30 03:09:28.185445 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:28.185205 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn" Oct 30 03:09:28.285062 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:28.281197 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed2b192-3c98-4542-a416-ce77621acf8d-kube-api-access-8dfks\" (UniqueName: \"kubernetes.io/projected/bed2b192-3c98-4542-a416-ce77621acf8d-kube-api-access-8dfks\") pod \"bed2b192-3c98-4542-a416-ce77621acf8d\" (UID: \"bed2b192-3c98-4542-a416-ce77621acf8d\") " Oct 30 03:09:28.285062 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:28.281291 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed2b192-3c98-4542-a416-ce77621acf8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed2b192-3c98-4542-a416-ce77621acf8d-maas-api-openshift-ca\") pod \"bed2b192-3c98-4542-a416-ce77621acf8d\" (UID: \"bed2b192-3c98-4542-a416-ce77621acf8d\") " Oct 30 03:09:28.285062 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:28.281438 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed2b192-3c98-4542-a416-ce77621acf8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed2b192-3c98-4542-a416-ce77621acf8d-maas-api-xks-ca\") pod \"bed2b192-3c98-4542-a416-ce77621acf8d\" (UID: \"bed2b192-3c98-4542-a416-ce77621acf8d\") " Oct 30 03:09:28.285062 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:28.283871 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed2b192-3c98-4542-a416-ce77621acf8d-maas-api-openshift-ca" pod "bed2b192-3c98-4542-a416-ce77621acf8d" (UID: "bed2b192-3c98-4542-a416-ce77621acf8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:09:28.286000 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:28.285965 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed2b192-3c98-4542-a416-ce77621acf8d-maas-api-xks-ca" pod "bed2b192-3c98-4542-a416-ce77621acf8d" (UID: "bed2b192-3c98-4542-a416-ce77621acf8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:09:28.286304 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:28.286271 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed2b192-3c98-4542-a416-ce77621acf8d-kube-api-access-8dfks" pod "bed2b192-3c98-4542-a416-ce77621acf8d" (UID: "bed2b192-3c98-4542-a416-ce77621acf8d"). InnerVolumeSpecName "kube-api-access-8dfks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:09:28.383244 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:28.383201 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed2b192-3c98-4542-a416-ce77621acf8d-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:09:28.383244 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:28.383240 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dfks\" (UniqueName: \"kubernetes.io/projected/bed2b192-3c98-4542-a416-ce77621acf8d-kube-api-access-8dfks\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:09:28.383244 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:28.383250 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed2b192-3c98-4542-a416-ce77621acf8d-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:09:28.877902 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:28.877795 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn" event={"ID":"bed2b192-3c98-4542-a416-ce77621acf8d","Type":"ContainerDied","Data":"3db3b8852b2d1d5c6925e9579ad42ec7efe5b6c29de0496b8c80bcefef29dd7d"} Oct 30 03:09:28.877902 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:28.877882 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3db3b8852b2d1d5c6925e9579ad42ec7efe5b6c29de0496b8c80bcefef29dd7d" Oct 30 03:09:28.878214 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:28.877970 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4260dc98-cgkzn" Oct 30 03:09:35.117478 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.117438 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd"] Oct 30 03:09:35.117968 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.117886 5998 cpu_manager.go:414] "containerMap: removing container" podUID="bed2b192-3c98-4542-a416-ce77621acf8d" containerName="revoke-keys" Oct 30 03:09:35.117968 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.117897 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed2b192-3c98-4542-a416-ce77621acf8d" containerName="revoke-keys" Oct 30 03:09:35.118049 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.117981 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed2b192-3c98-4542-a416-ce77621acf8d" containerName="revoke-keys" Oct 30 03:09:35.121466 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.121433 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd" Oct 30 03:09:35.135379 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.135350 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:09:35.146958 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.146921 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:09:35.147163 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.146976 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:09:35.147163 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.147059 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:09:35.149562 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.149533 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d758cf-0d42-4c64-af14-6a545c14280a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b0b1dc5-m9gjd\" (UID: \"d3d758cf-0d42-4c64-af14-6a545c14280a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd" Oct 30 03:09:35.149722 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.149614 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d758cf-0d42-4c64-af14-6a545c14280a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b0b1dc5-m9gjd\" (UID: \"d3d758cf-0d42-4c64-af14-6a545c14280a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd" Oct 30 03:09:35.149722 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.149685 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxqx2\" (UniqueName: \"kubernetes.io/projected/d3d758cf-0d42-4c64-af14-6a545c14280a-kube-api-access-fxqx2\") pod \"maas-api-revoke-sub-0b0b1dc5-m9gjd\" (UID: \"d3d758cf-0d42-4c64-af14-6a545c14280a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd" Oct 30 03:09:35.150236 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.150208 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd"] Oct 30 03:09:35.250412 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.250363 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d758cf-0d42-4c64-af14-6a545c14280a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b0b1dc5-m9gjd\" (UID: \"d3d758cf-0d42-4c64-af14-6a545c14280a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd" Oct 30 03:09:35.250634 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.250500 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d758cf-0d42-4c64-af14-6a545c14280a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b0b1dc5-m9gjd\" (UID: \"d3d758cf-0d42-4c64-af14-6a545c14280a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd" Oct 30 03:09:35.250634 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.250541 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxqx2\" (UniqueName: \"kubernetes.io/projected/d3d758cf-0d42-4c64-af14-6a545c14280a-kube-api-access-fxqx2\") pod \"maas-api-revoke-sub-0b0b1dc5-m9gjd\" (UID: \"d3d758cf-0d42-4c64-af14-6a545c14280a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd" Oct 30 03:09:35.251261 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.251239 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d758cf-0d42-4c64-af14-6a545c14280a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b0b1dc5-m9gjd\" (UID: \"d3d758cf-0d42-4c64-af14-6a545c14280a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd" Oct 30 03:09:35.252756 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.252735 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d758cf-0d42-4c64-af14-6a545c14280a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b0b1dc5-m9gjd\" (UID: \"d3d758cf-0d42-4c64-af14-6a545c14280a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd" Oct 30 03:09:35.273659 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.273627 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxqx2\" (UniqueName: \"kubernetes.io/projected/d3d758cf-0d42-4c64-af14-6a545c14280a-kube-api-access-fxqx2\") pod \"maas-api-revoke-sub-0b0b1dc5-m9gjd\" (UID: \"d3d758cf-0d42-4c64-af14-6a545c14280a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd" Oct 30 03:09:35.435177 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.435145 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd" Oct 30 03:09:35.741614 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.741533 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd"] Oct 30 03:09:35.745898 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:09:35.745862 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d758cf_0d42_4c64_af14_6a545c14280a.slice/crio-f48d413302946644a3725b2907088cabb049208d74f8380bc32f19817ea82a7d WatchSource:0}: Error finding container f48d413302946644a3725b2907088cabb049208d74f8380bc32f19817ea82a7d: Status 404 returned error can't find the container with id f48d413302946644a3725b2907088cabb049208d74f8380bc32f19817ea82a7d Oct 30 03:09:35.919286 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:35.919246 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd" event={"ID":"d3d758cf-0d42-4c64-af14-6a545c14280a","Type":"ContainerStarted","Data":"f48d413302946644a3725b2907088cabb049208d74f8380bc32f19817ea82a7d"} Oct 30 03:09:36.929793 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:36.929748 5998 generic.go:356] "Generic (PLEG): container finished" podID="d3d758cf-0d42-4c64-af14-6a545c14280a" containerID="3bad45b70dbf7d98aa4860d50ddedf19e98a6cf55724b3d7f295a7ac5dd60956" exitCode=0 Oct 30 03:09:36.930377 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:36.929822 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd" event={"ID":"d3d758cf-0d42-4c64-af14-6a545c14280a","Type":"ContainerDied","Data":"3bad45b70dbf7d98aa4860d50ddedf19e98a6cf55724b3d7f295a7ac5dd60956"} Oct 30 03:09:38.483960 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:38.483570 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd" Oct 30 03:09:38.527158 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:38.526629 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d758cf-0d42-4c64-af14-6a545c14280a-kube-api-access-fxqx2\" (UniqueName: \"kubernetes.io/projected/d3d758cf-0d42-4c64-af14-6a545c14280a-kube-api-access-fxqx2\") pod \"d3d758cf-0d42-4c64-af14-6a545c14280a\" (UID: \"d3d758cf-0d42-4c64-af14-6a545c14280a\") " Oct 30 03:09:38.527472 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:38.527448 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d758cf-0d42-4c64-af14-6a545c14280a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d758cf-0d42-4c64-af14-6a545c14280a-maas-api-openshift-ca\") pod \"d3d758cf-0d42-4c64-af14-6a545c14280a\" (UID: \"d3d758cf-0d42-4c64-af14-6a545c14280a\") " Oct 30 03:09:38.565403 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:38.565370 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3d758cf-0d42-4c64-af14-6a545c14280a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d758cf-0d42-4c64-af14-6a545c14280a-maas-api-xks-ca\") pod \"d3d758cf-0d42-4c64-af14-6a545c14280a\" (UID: \"d3d758cf-0d42-4c64-af14-6a545c14280a\") " Oct 30 03:09:38.566064 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:38.564815 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d758cf-0d42-4c64-af14-6a545c14280a-maas-api-openshift-ca" pod "d3d758cf-0d42-4c64-af14-6a545c14280a" (UID: "d3d758cf-0d42-4c64-af14-6a545c14280a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:09:38.566651 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:38.566625 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d758cf-0d42-4c64-af14-6a545c14280a-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:09:38.567601 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:38.566905 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d758cf-0d42-4c64-af14-6a545c14280a-kube-api-access-fxqx2" pod "d3d758cf-0d42-4c64-af14-6a545c14280a" (UID: "d3d758cf-0d42-4c64-af14-6a545c14280a"). InnerVolumeSpecName "kube-api-access-fxqx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:09:38.582471 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:38.582340 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3d758cf-0d42-4c64-af14-6a545c14280a-maas-api-xks-ca" pod "d3d758cf-0d42-4c64-af14-6a545c14280a" (UID: "d3d758cf-0d42-4c64-af14-6a545c14280a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:09:38.668974 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:38.668867 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d758cf-0d42-4c64-af14-6a545c14280a-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:09:38.668974 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:38.668923 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxqx2\" (UniqueName: \"kubernetes.io/projected/d3d758cf-0d42-4c64-af14-6a545c14280a-kube-api-access-fxqx2\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:09:38.975170 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:38.973370 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd" event={"ID":"d3d758cf-0d42-4c64-af14-6a545c14280a","Type":"ContainerDied","Data":"f48d413302946644a3725b2907088cabb049208d74f8380bc32f19817ea82a7d"} Oct 30 03:09:38.975170 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:38.973435 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f48d413302946644a3725b2907088cabb049208d74f8380bc32f19817ea82a7d" Oct 30 03:09:38.975170 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:38.973447 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b0b1dc5-m9gjd" Oct 30 03:09:44.751154 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:44.751114 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:09:45.461104 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.461055 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5"] Oct 30 03:09:45.461489 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.461476 5998 cpu_manager.go:414] "containerMap: removing container" podUID="d3d758cf-0d42-4c64-af14-6a545c14280a" containerName="revoke-keys" Oct 30 03:09:45.461537 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.461490 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d758cf-0d42-4c64-af14-6a545c14280a" containerName="revoke-keys" Oct 30 03:09:45.461584 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.461567 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d758cf-0d42-4c64-af14-6a545c14280a" containerName="revoke-keys" Oct 30 03:09:45.464757 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.464737 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5" Oct 30 03:09:45.478748 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.478718 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:09:45.490435 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.490398 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:09:45.490616 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.490439 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:09:45.494002 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.493976 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5"] Oct 30 03:09:45.533298 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.533267 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab93e370-1050-4495-9e42-dabebabaf642-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4230a525-kdjc5\" (UID: \"ab93e370-1050-4495-9e42-dabebabaf642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5" Oct 30 03:09:45.533475 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.533310 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab93e370-1050-4495-9e42-dabebabaf642-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4230a525-kdjc5\" (UID: \"ab93e370-1050-4495-9e42-dabebabaf642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5" Oct 30 03:09:45.533475 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.533442 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qhzj\" (UniqueName: \"kubernetes.io/projected/ab93e370-1050-4495-9e42-dabebabaf642-kube-api-access-5qhzj\") pod \"maas-api-revoke-sub-4230a525-kdjc5\" (UID: \"ab93e370-1050-4495-9e42-dabebabaf642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5" Oct 30 03:09:45.579548 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.579511 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:09:45.634424 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.634388 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab93e370-1050-4495-9e42-dabebabaf642-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4230a525-kdjc5\" (UID: \"ab93e370-1050-4495-9e42-dabebabaf642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5" Oct 30 03:09:45.634649 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.634559 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab93e370-1050-4495-9e42-dabebabaf642-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4230a525-kdjc5\" (UID: \"ab93e370-1050-4495-9e42-dabebabaf642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5" Oct 30 03:09:45.634839 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.634797 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qhzj\" (UniqueName: \"kubernetes.io/projected/ab93e370-1050-4495-9e42-dabebabaf642-kube-api-access-5qhzj\") pod \"maas-api-revoke-sub-4230a525-kdjc5\" (UID: \"ab93e370-1050-4495-9e42-dabebabaf642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5" Oct 30 03:09:45.635656 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.635383 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab93e370-1050-4495-9e42-dabebabaf642-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4230a525-kdjc5\" (UID: \"ab93e370-1050-4495-9e42-dabebabaf642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5" Oct 30 03:09:45.637184 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.637159 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab93e370-1050-4495-9e42-dabebabaf642-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4230a525-kdjc5\" (UID: \"ab93e370-1050-4495-9e42-dabebabaf642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5" Oct 30 03:09:45.657756 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.657729 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qhzj\" (UniqueName: \"kubernetes.io/projected/ab93e370-1050-4495-9e42-dabebabaf642-kube-api-access-5qhzj\") pod \"maas-api-revoke-sub-4230a525-kdjc5\" (UID: \"ab93e370-1050-4495-9e42-dabebabaf642\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5" Oct 30 03:09:45.778807 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:45.778719 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5" Oct 30 03:09:46.081293 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:46.081208 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5"] Oct 30 03:09:46.084794 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:09:46.084755 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab93e370_1050_4495_9e42_dabebabaf642.slice/crio-4f2315b7382127242a4d5e27772086c85cd4fb5e196f7aa6b95d80ee6792b959 WatchSource:0}: Error finding container 4f2315b7382127242a4d5e27772086c85cd4fb5e196f7aa6b95d80ee6792b959: Status 404 returned error can't find the container with id 4f2315b7382127242a4d5e27772086c85cd4fb5e196f7aa6b95d80ee6792b959 Oct 30 03:09:47.024923 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:47.024878 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5" event={"ID":"ab93e370-1050-4495-9e42-dabebabaf642","Type":"ContainerStarted","Data":"f9f609f705f3a969eedd7859cdd2b4a9f15921bd6a89f481fad92c3ba30532fb"} Oct 30 03:09:47.025385 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:47.024953 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5" event={"ID":"ab93e370-1050-4495-9e42-dabebabaf642","Type":"ContainerStarted","Data":"4f2315b7382127242a4d5e27772086c85cd4fb5e196f7aa6b95d80ee6792b959"} Oct 30 03:09:47.071085 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:47.071019 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5" podStartSLOduration=2.071001448 podStartE2EDuration="2.071001448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:47.068528883 +0000 UTC m=+6016.647319966" watchObservedRunningTime="2026-10-30 03:09:47.071001448 +0000 UTC m=+6016.649792530" Oct 30 03:09:48.070116 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:48.069084 5998 generic.go:356] "Generic (PLEG): container finished" podID="ab93e370-1050-4495-9e42-dabebabaf642" containerID="f9f609f705f3a969eedd7859cdd2b4a9f15921bd6a89f481fad92c3ba30532fb" exitCode=0 Oct 30 03:09:48.070116 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:48.069256 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5" event={"ID":"ab93e370-1050-4495-9e42-dabebabaf642","Type":"ContainerDied","Data":"f9f609f705f3a969eedd7859cdd2b4a9f15921bd6a89f481fad92c3ba30532fb"} Oct 30 03:09:49.435011 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:49.434976 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5" Oct 30 03:09:49.580908 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:49.580862 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab93e370-1050-4495-9e42-dabebabaf642-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab93e370-1050-4495-9e42-dabebabaf642-maas-api-xks-ca\") pod \"ab93e370-1050-4495-9e42-dabebabaf642\" (UID: \"ab93e370-1050-4495-9e42-dabebabaf642\") " Oct 30 03:09:49.581153 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:49.581124 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab93e370-1050-4495-9e42-dabebabaf642-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab93e370-1050-4495-9e42-dabebabaf642-maas-api-openshift-ca\") pod \"ab93e370-1050-4495-9e42-dabebabaf642\" (UID: \"ab93e370-1050-4495-9e42-dabebabaf642\") " Oct 30 03:09:49.581255 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:49.581195 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab93e370-1050-4495-9e42-dabebabaf642-kube-api-access-5qhzj\" (UniqueName: \"kubernetes.io/projected/ab93e370-1050-4495-9e42-dabebabaf642-kube-api-access-5qhzj\") pod \"ab93e370-1050-4495-9e42-dabebabaf642\" (UID: \"ab93e370-1050-4495-9e42-dabebabaf642\") " Oct 30 03:09:49.581648 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:49.581617 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab93e370-1050-4495-9e42-dabebabaf642-maas-api-openshift-ca" pod "ab93e370-1050-4495-9e42-dabebabaf642" (UID: "ab93e370-1050-4495-9e42-dabebabaf642"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:09:49.583852 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:49.583814 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab93e370-1050-4495-9e42-dabebabaf642-kube-api-access-5qhzj" pod "ab93e370-1050-4495-9e42-dabebabaf642" (UID: "ab93e370-1050-4495-9e42-dabebabaf642"). InnerVolumeSpecName "kube-api-access-5qhzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:09:49.587224 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:49.587192 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab93e370-1050-4495-9e42-dabebabaf642-maas-api-xks-ca" pod "ab93e370-1050-4495-9e42-dabebabaf642" (UID: "ab93e370-1050-4495-9e42-dabebabaf642"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:09:49.681966 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:49.681909 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qhzj\" (UniqueName: \"kubernetes.io/projected/ab93e370-1050-4495-9e42-dabebabaf642-kube-api-access-5qhzj\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:09:49.681966 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:49.681949 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab93e370-1050-4495-9e42-dabebabaf642-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:09:49.681966 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:49.681966 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab93e370-1050-4495-9e42-dabebabaf642-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:09:50.088616 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:50.088529 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5" event={"ID":"ab93e370-1050-4495-9e42-dabebabaf642","Type":"ContainerDied","Data":"4f2315b7382127242a4d5e27772086c85cd4fb5e196f7aa6b95d80ee6792b959"} Oct 30 03:09:50.088616 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:50.088555 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4230a525-kdjc5" Oct 30 03:09:50.088616 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:50.088583 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f2315b7382127242a4d5e27772086c85cd4fb5e196f7aa6b95d80ee6792b959" Oct 30 03:09:55.215902 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:09:55.215865 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:09:55.815995 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.815948 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7"] Oct 30 03:09:55.816436 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.816418 5998 cpu_manager.go:414] "containerMap: removing container" podUID="ab93e370-1050-4495-9e42-dabebabaf642" containerName="revoke-keys" Oct 30 03:09:55.816436 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.816435 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab93e370-1050-4495-9e42-dabebabaf642" containerName="revoke-keys" Oct 30 03:09:55.816621 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.816522 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab93e370-1050-4495-9e42-dabebabaf642" containerName="revoke-keys" Oct 30 03:09:55.819449 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.819424 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7" Oct 30 03:09:55.833443 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.833401 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:09:55.841948 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.841917 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556e1aee-83d0-41a5-9393-b08d28be8248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f604db78-jkvz7\" (UID: \"556e1aee-83d0-41a5-9393-b08d28be8248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7" Oct 30 03:09:55.842120 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.841963 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmsmb\" (UniqueName: \"kubernetes.io/projected/556e1aee-83d0-41a5-9393-b08d28be8248-kube-api-access-vmsmb\") pod \"maas-api-revoke-sub-f604db78-jkvz7\" (UID: \"556e1aee-83d0-41a5-9393-b08d28be8248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7" Oct 30 03:09:55.842231 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.842202 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556e1aee-83d0-41a5-9393-b08d28be8248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f604db78-jkvz7\" (UID: \"556e1aee-83d0-41a5-9393-b08d28be8248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7" Oct 30 03:09:55.845146 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.845118 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:09:55.845319 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.845175 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:09:55.845319 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.845179 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:09:55.849312 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.849282 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7"] Oct 30 03:09:55.943362 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.943321 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556e1aee-83d0-41a5-9393-b08d28be8248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f604db78-jkvz7\" (UID: \"556e1aee-83d0-41a5-9393-b08d28be8248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7" Oct 30 03:09:55.943553 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.943408 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556e1aee-83d0-41a5-9393-b08d28be8248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f604db78-jkvz7\" (UID: \"556e1aee-83d0-41a5-9393-b08d28be8248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7" Oct 30 03:09:55.943553 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.943439 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmsmb\" (UniqueName: \"kubernetes.io/projected/556e1aee-83d0-41a5-9393-b08d28be8248-kube-api-access-vmsmb\") pod \"maas-api-revoke-sub-f604db78-jkvz7\" (UID: \"556e1aee-83d0-41a5-9393-b08d28be8248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7" Oct 30 03:09:55.944179 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.944141 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556e1aee-83d0-41a5-9393-b08d28be8248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f604db78-jkvz7\" (UID: \"556e1aee-83d0-41a5-9393-b08d28be8248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7" Oct 30 03:09:55.945937 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.945907 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556e1aee-83d0-41a5-9393-b08d28be8248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f604db78-jkvz7\" (UID: \"556e1aee-83d0-41a5-9393-b08d28be8248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7" Oct 30 03:09:55.966329 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:55.966300 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmsmb\" (UniqueName: \"kubernetes.io/projected/556e1aee-83d0-41a5-9393-b08d28be8248-kube-api-access-vmsmb\") pod \"maas-api-revoke-sub-f604db78-jkvz7\" (UID: \"556e1aee-83d0-41a5-9393-b08d28be8248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7" Oct 30 03:09:56.132818 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:56.132717 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7" Oct 30 03:09:56.433351 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:56.433309 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7"] Oct 30 03:09:56.436847 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:09:56.436813 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556e1aee_83d0_41a5_9393_b08d28be8248.slice/crio-612e40cf34b82dbfbb1130931b4f4ed748f42272686baac4f84e63fc492eeec1 WatchSource:0}: Error finding container 612e40cf34b82dbfbb1130931b4f4ed748f42272686baac4f84e63fc492eeec1: Status 404 returned error can't find the container with id 612e40cf34b82dbfbb1130931b4f4ed748f42272686baac4f84e63fc492eeec1 Oct 30 03:09:57.130015 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:57.129914 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7" event={"ID":"556e1aee-83d0-41a5-9393-b08d28be8248","Type":"ContainerStarted","Data":"46a898f2bcec4aa9950d5892121db99d32d9fc1a83b526b55a73980419d1b159"} Oct 30 03:09:57.130015 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:57.129969 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7" event={"ID":"556e1aee-83d0-41a5-9393-b08d28be8248","Type":"ContainerStarted","Data":"612e40cf34b82dbfbb1130931b4f4ed748f42272686baac4f84e63fc492eeec1"} Oct 30 03:09:57.174358 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:57.174302 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7" podStartSLOduration=2.174286718 podStartE2EDuration="2.174286718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:57.172628266 +0000 UTC m=+6026.751419347" watchObservedRunningTime="2026-10-30 03:09:57.174286718 +0000 UTC m=+6026.753077858" Oct 30 03:09:58.148001 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:58.147511 5998 generic.go:356] "Generic (PLEG): container finished" podID="556e1aee-83d0-41a5-9393-b08d28be8248" containerID="46a898f2bcec4aa9950d5892121db99d32d9fc1a83b526b55a73980419d1b159" exitCode=0 Oct 30 03:09:58.148001 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:58.147572 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7" event={"ID":"556e1aee-83d0-41a5-9393-b08d28be8248","Type":"ContainerDied","Data":"46a898f2bcec4aa9950d5892121db99d32d9fc1a83b526b55a73980419d1b159"} Oct 30 03:09:59.499701 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:59.499673 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7" Oct 30 03:09:59.666115 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:59.665415 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/556e1aee-83d0-41a5-9393-b08d28be8248-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556e1aee-83d0-41a5-9393-b08d28be8248-maas-api-openshift-ca\") pod \"556e1aee-83d0-41a5-9393-b08d28be8248\" (UID: \"556e1aee-83d0-41a5-9393-b08d28be8248\") " Oct 30 03:09:59.666115 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:59.665532 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/556e1aee-83d0-41a5-9393-b08d28be8248-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556e1aee-83d0-41a5-9393-b08d28be8248-maas-api-xks-ca\") pod \"556e1aee-83d0-41a5-9393-b08d28be8248\" (UID: \"556e1aee-83d0-41a5-9393-b08d28be8248\") " Oct 30 03:09:59.666115 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:59.665622 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/556e1aee-83d0-41a5-9393-b08d28be8248-kube-api-access-vmsmb\" (UniqueName: \"kubernetes.io/projected/556e1aee-83d0-41a5-9393-b08d28be8248-kube-api-access-vmsmb\") pod \"556e1aee-83d0-41a5-9393-b08d28be8248\" (UID: \"556e1aee-83d0-41a5-9393-b08d28be8248\") " Oct 30 03:09:59.668968 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:59.667417 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/556e1aee-83d0-41a5-9393-b08d28be8248-maas-api-openshift-ca" pod "556e1aee-83d0-41a5-9393-b08d28be8248" (UID: "556e1aee-83d0-41a5-9393-b08d28be8248"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:09:59.671790 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:59.671750 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/556e1aee-83d0-41a5-9393-b08d28be8248-maas-api-xks-ca" pod "556e1aee-83d0-41a5-9393-b08d28be8248" (UID: "556e1aee-83d0-41a5-9393-b08d28be8248"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:09:59.672638 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:59.672599 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556e1aee-83d0-41a5-9393-b08d28be8248-kube-api-access-vmsmb" pod "556e1aee-83d0-41a5-9393-b08d28be8248" (UID: "556e1aee-83d0-41a5-9393-b08d28be8248"). InnerVolumeSpecName "kube-api-access-vmsmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:09:59.766790 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:59.766626 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556e1aee-83d0-41a5-9393-b08d28be8248-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:09:59.766790 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:59.766674 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556e1aee-83d0-41a5-9393-b08d28be8248-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:09:59.766790 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:09:59.766691 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmsmb\" (UniqueName: \"kubernetes.io/projected/556e1aee-83d0-41a5-9393-b08d28be8248-kube-api-access-vmsmb\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:10:00.176673 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:00.176637 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7" event={"ID":"556e1aee-83d0-41a5-9393-b08d28be8248","Type":"ContainerDied","Data":"612e40cf34b82dbfbb1130931b4f4ed748f42272686baac4f84e63fc492eeec1"} Oct 30 03:10:00.176866 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:00.176701 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="612e40cf34b82dbfbb1130931b4f4ed748f42272686baac4f84e63fc492eeec1" Oct 30 03:10:00.176866 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:00.176646 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f604db78-jkvz7" Oct 30 03:10:06.140051 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.140012 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj"] Oct 30 03:10:06.140469 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.140453 5998 cpu_manager.go:414] "containerMap: removing container" podUID="556e1aee-83d0-41a5-9393-b08d28be8248" containerName="revoke-keys" Oct 30 03:10:06.140532 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.140470 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="556e1aee-83d0-41a5-9393-b08d28be8248" containerName="revoke-keys" Oct 30 03:10:06.140576 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.140544 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="556e1aee-83d0-41a5-9393-b08d28be8248" containerName="revoke-keys" Oct 30 03:10:06.144218 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.144185 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj" Oct 30 03:10:06.158397 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.158360 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:10:06.158397 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.158360 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 03:10:06.170049 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.170014 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:10:06.170049 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.170020 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-z9rqp\"" Oct 30 03:10:06.173863 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.173830 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj"] Oct 30 03:10:06.334626 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.334572 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fb66158-bkxmj\" (UID: \"cfdd2ee5-7258-4144-a440-6b085d1c2a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj" Oct 30 03:10:06.334867 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.334730 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fb66158-bkxmj\" (UID: \"cfdd2ee5-7258-4144-a440-6b085d1c2a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj" Oct 30 03:10:06.334867 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.334796 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk6d4\" (UniqueName: \"kubernetes.io/projected/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-kube-api-access-gk6d4\") pod \"maas-api-revoke-sub-3fb66158-bkxmj\" (UID: \"cfdd2ee5-7258-4144-a440-6b085d1c2a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj" Oct 30 03:10:06.435578 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.435525 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fb66158-bkxmj\" (UID: \"cfdd2ee5-7258-4144-a440-6b085d1c2a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj" Oct 30 03:10:06.435811 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.435589 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk6d4\" (UniqueName: \"kubernetes.io/projected/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-kube-api-access-gk6d4\") pod \"maas-api-revoke-sub-3fb66158-bkxmj\" (UID: \"cfdd2ee5-7258-4144-a440-6b085d1c2a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj" Oct 30 03:10:06.435811 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.435624 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fb66158-bkxmj\" (UID: \"cfdd2ee5-7258-4144-a440-6b085d1c2a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj" Oct 30 03:10:06.436375 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.436348 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fb66158-bkxmj\" (UID: \"cfdd2ee5-7258-4144-a440-6b085d1c2a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj" Oct 30 03:10:06.438064 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.438031 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fb66158-bkxmj\" (UID: \"cfdd2ee5-7258-4144-a440-6b085d1c2a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj" Oct 30 03:10:06.461352 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.461318 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk6d4\" (UniqueName: \"kubernetes.io/projected/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-kube-api-access-gk6d4\") pod \"maas-api-revoke-sub-3fb66158-bkxmj\" (UID: \"cfdd2ee5-7258-4144-a440-6b085d1c2a4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj" Oct 30 03:10:06.758632 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:06.758537 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj" Oct 30 03:10:07.071912 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:07.071818 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj"] Oct 30 03:10:07.075316 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:10:07.075285 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfdd2ee5_7258_4144_a440_6b085d1c2a4e.slice/crio-70ceaab0e687d1c421728904b5fdba26453270d11af0e1a849d759c22bf294cd WatchSource:0}: Error finding container 70ceaab0e687d1c421728904b5fdba26453270d11af0e1a849d759c22bf294cd: Status 404 returned error can't find the container with id 70ceaab0e687d1c421728904b5fdba26453270d11af0e1a849d759c22bf294cd Oct 30 03:10:07.224369 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:07.224326 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj" event={"ID":"cfdd2ee5-7258-4144-a440-6b085d1c2a4e","Type":"ContainerStarted","Data":"70ceaab0e687d1c421728904b5fdba26453270d11af0e1a849d759c22bf294cd"} Oct 30 03:10:08.271201 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:08.262730 5998 generic.go:356] "Generic (PLEG): container finished" podID="cfdd2ee5-7258-4144-a440-6b085d1c2a4e" containerID="67e395d3567f10420c81408b997cfb2cbd7a80d14506f234921e548e003a57ff" exitCode=0 Oct 30 03:10:08.271201 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:08.262895 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj" event={"ID":"cfdd2ee5-7258-4144-a440-6b085d1c2a4e","Type":"ContainerDied","Data":"67e395d3567f10420c81408b997cfb2cbd7a80d14506f234921e548e003a57ff"} Oct 30 03:10:09.223845 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:09.223803 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:10:09.556225 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:09.556196 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj" Oct 30 03:10:09.592188 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:09.592142 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-maas-api-xks-ca\") pod \"cfdd2ee5-7258-4144-a440-6b085d1c2a4e\" (UID: \"cfdd2ee5-7258-4144-a440-6b085d1c2a4e\") " Oct 30 03:10:09.592414 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:09.592256 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-maas-api-openshift-ca\") pod \"cfdd2ee5-7258-4144-a440-6b085d1c2a4e\" (UID: \"cfdd2ee5-7258-4144-a440-6b085d1c2a4e\") " Oct 30 03:10:09.592414 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:09.592300 5998 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-kube-api-access-gk6d4\" (UniqueName: \"kubernetes.io/projected/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-kube-api-access-gk6d4\") pod \"cfdd2ee5-7258-4144-a440-6b085d1c2a4e\" (UID: \"cfdd2ee5-7258-4144-a440-6b085d1c2a4e\") " Oct 30 03:10:09.593348 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:09.593302 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-maas-api-openshift-ca" pod "cfdd2ee5-7258-4144-a440-6b085d1c2a4e" (UID: "cfdd2ee5-7258-4144-a440-6b085d1c2a4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:10:09.594956 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:09.594922 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-maas-api-xks-ca" pod "cfdd2ee5-7258-4144-a440-6b085d1c2a4e" (UID: "cfdd2ee5-7258-4144-a440-6b085d1c2a4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:10:09.595564 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:09.595531 5998 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-kube-api-access-gk6d4" pod "cfdd2ee5-7258-4144-a440-6b085d1c2a4e" (UID: "cfdd2ee5-7258-4144-a440-6b085d1c2a4e"). InnerVolumeSpecName "kube-api-access-gk6d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:10:09.693407 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:09.693366 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-maas-api-xks-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:10:09.693407 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:09.693403 5998 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-maas-api-openshift-ca\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:10:09.693407 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:09.693413 5998 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk6d4\" (UniqueName: \"kubernetes.io/projected/cfdd2ee5-7258-4144-a440-6b085d1c2a4e-kube-api-access-gk6d4\") on node \"ip-10-0-130-0.ec2.internal\" DevicePath \"\"" Oct 30 03:10:10.289327 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:10.289292 5998 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj" Oct 30 03:10:10.289516 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:10.289322 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fb66158-bkxmj" event={"ID":"cfdd2ee5-7258-4144-a440-6b085d1c2a4e","Type":"ContainerDied","Data":"70ceaab0e687d1c421728904b5fdba26453270d11af0e1a849d759c22bf294cd"} Oct 30 03:10:10.289516 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:10.289377 5998 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70ceaab0e687d1c421728904b5fdba26453270d11af0e1a849d759c22bf294cd" Oct 30 03:10:17.478246 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:17.478154 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p2m8w"] Oct 30 03:10:25.218153 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:10:25.218112 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:10:34.156791 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:34.156755 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:10:41.404736 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:41.404705 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-25r9k_7994cb01-5d42-4871-ae17-d9622123f971/manager/0.log" Oct 30 03:10:41.608531 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:41.608498 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zr7w4_114ffe72-9ce9-4eed-baea-d3667e969148/manager/0.log" Oct 30 03:10:41.840492 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:41.840456 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-wptrw_1a3c5537-f83e-412e-a917-27fd46cfe258/manager/0.log" Oct 30 03:10:42.381108 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:42.381054 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9fjjh_712fc2f5-d537-4e96-9801-963522329de7/server/0.log" Oct 30 03:10:42.597186 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:42.597149 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kpkmz_04727d53-1a8f-4f08-9930-acfb77f05eb4/manager/0.log" Oct 30 03:10:43.034073 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:43.034032 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8"] Oct 30 03:10:43.037552 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:43.037526 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f782de00-fvvm8"] Oct 30 03:10:44.457225 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:44.457198 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0b0b1dc5-m9gjd_d3d758cf-0d42-4c64-af14-6a545c14280a/revoke-keys/0.log" Oct 30 03:10:44.543795 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:44.543753 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ab4fe3-71c1-4d24-93a9-6c0686ef559b" path="/var/lib/kubelet/pods/48ab4fe3-71c1-4d24-93a9-6c0686ef559b/volumes" Oct 30 03:10:44.663352 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:44.663319 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-34ae66f0-rpxnr_d7ebac2a-738c-4851-904d-c752b06d5501/revoke-keys/0.log" Oct 30 03:10:44.849467 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:44.849373 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:10:44.852439 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:44.852416 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3fb66158-bkxmj_cfdd2ee5-7258-4144-a440-6b085d1c2a4e/revoke-keys/0.log" Oct 30 03:10:45.046549 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:45.046516 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4230a525-kdjc5_ab93e370-1050-4495-9e42-dabebabaf642/revoke-keys/0.log" Oct 30 03:10:45.233286 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:45.233254 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4260dc98-cgkzn_bed2b192-3c98-4542-a416-ce77621acf8d/revoke-keys/0.log" Oct 30 03:10:45.446777 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:45.446748 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-485ab6cf-kv7pb_e56e09aa-2161-4528-9238-773299002451/revoke-keys/0.log" Oct 30 03:10:45.642750 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:45.642665 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4ff3b3d0-bkzz2_c6d8e235-ccbe-420e-b07b-5c3f96252538/revoke-keys/0.log" Oct 30 03:10:45.839186 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:45.839152 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5329dbbc-wrr46_8666bad4-75eb-4159-bd56-8a194a3ff6a8/revoke-keys/0.log" Oct 30 03:10:46.040362 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:46.040336 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-590d399f-2kwtl_825aabe5-05da-4610-a8e4-c312ba9d1327/revoke-keys/0.log" Oct 30 03:10:46.242166 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:46.242132 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6da419fc-dqrx8_405d89d5-4a2c-4319-bffb-20e2bca0f82f/revoke-keys/0.log" Oct 30 03:10:46.426067 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:46.426034 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6f2d194d-kjpd7_dd243323-498d-41c8-a63a-05380de91423/revoke-keys/0.log" Oct 30 03:10:46.642194 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:46.642162 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9338dd48-n5bf8_2fe8a28d-5d3e-447c-80f1-f3c9ef91cf69/revoke-keys/0.log" Oct 30 03:10:46.858447 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:46.858357 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9d4ef071-r5z4k_d70785ea-4edd-466b-99e9-dba594675a9e/revoke-keys/0.log" Oct 30 03:10:47.050068 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:47.050040 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ac2f8859-l2b69_f3fac7d0-7f99-4b09-bbbc-3b32497a67ee/revoke-keys/0.log" Oct 30 03:10:47.242606 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:47.242573 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b1ce96eb-v7fhp_9bd6af8d-fd7f-4715-bcf9-f3eab66a1f90/revoke-keys/0.log" Oct 30 03:10:47.448627 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:47.448491 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bde2c551-gs8hc_2126872e-0a4b-4d18-8985-d1aeb474052b/revoke-keys/0.log" Oct 30 03:10:47.637909 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:47.637834 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c10fe630-w2mvg_e2e0e937-3a12-45c1-bbb6-ef31130c76ee/revoke-keys/0.log" Oct 30 03:10:47.835261 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:47.835227 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f604db78-jkvz7_556e1aee-83d0-41a5-9393-b08d28be8248/revoke-keys/0.log" Oct 30 03:10:50.393145 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:50.393113 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-z4b6g_db689d73-bcb1-41fb-8507-84109c764878/manager/0.log" Oct 30 03:10:50.827345 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:50.827306 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-s228v_6d3c1f40-053d-4b00-83a2-a6308a90a0b8/manager/0.log" Oct 30 03:10:51.105201 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:51.105123 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-p2m8w_db342fa8-f998-43b9-8e0a-5da7612cdee2/limitador/0.log" Oct 30 03:10:52.995027 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:52.994993 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bd6fc9ff9-w5zwp_746a1452-89c9-4b38-9767-318905eaf8fe/kube-auth-proxy/0.log" Oct 30 03:10:53.190349 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:53.190318 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-28ssd_d8da8b22-5586-419c-9378-03d3d2d2c47f/istio-proxy/0.log" Oct 30 03:10:53.808042 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:53.808004 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-b95w4_0e843d95-ed87-4ea3-9ce8-d3f0103d9a80/istio-proxy/0.log" Oct 30 03:10:54.036048 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:54.036006 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr"] Oct 30 03:10:54.047331 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:54.047282 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ae66f0-rpxnr"] Oct 30 03:10:54.537572 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:54.537535 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7ebac2a-738c-4851-904d-c752b06d5501" path="/var/lib/kubelet/pods/d7ebac2a-738c-4851-904d-c752b06d5501/volumes" Oct 30 03:10:54.654335 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:54.654301 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-ppvtr_961683fb-469d-4d4a-9463-269eb264251a/istio-proxy/0.log" Oct 30 03:10:55.218131 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:10:55.218076 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:10:55.315927 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:10:55.315894 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q_b0a8f11c-0b47-448d-9c4a-d98899aabf93/istio-proxy/0.log" Oct 30 03:11:09.675513 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:09.675469 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:09.815922 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:09.815880 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:15.027006 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:15.026968 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-55kfb_e0bbf87d-7baa-4b99-9b80-695c2c99c4c8/global-pull-secret-syncer/0.log" Oct 30 03:11:15.898937 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:15.898899 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" Oct 30 03:11:16.035405 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:16.035367 5998 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69"] Oct 30 03:11:16.046817 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:16.046785 5998 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac2f8859-l2b69"] Oct 30 03:11:16.374013 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:16.373978 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-0.ec2.internal_8da95128540617dd483ff23c5d0ddd95/haproxy/0.log" Oct 30 03:11:16.537777 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:16.537743 5998 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3fac7d0-7f99-4b09-bbbc-3b32497a67ee" path="/var/lib/kubelet/pods/f3fac7d0-7f99-4b09-bbbc-3b32497a67ee/volumes" Oct 30 03:11:17.595617 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:17.595581 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-25r9k_7994cb01-5d42-4871-ae17-d9622123f971/manager/0.log" Oct 30 03:11:17.717406 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:17.717354 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zr7w4_114ffe72-9ce9-4eed-baea-d3667e969148/manager/0.log" Oct 30 03:11:18.465132 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:18.465084 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-wptrw_1a3c5537-f83e-412e-a917-27fd46cfe258/manager/0.log" Oct 30 03:11:18.731268 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:18.731181 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9fjjh_712fc2f5-d537-4e96-9801-963522329de7/server/0.log" Oct 30 03:11:18.875575 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:18.875535 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kpkmz_04727d53-1a8f-4f08-9930-acfb77f05eb4/manager/0.log" Oct 30 03:11:20.419584 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:20.419530 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-25r9k_7994cb01-5d42-4871-ae17-d9622123f971/manager/0.log" Oct 30 03:11:20.540011 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:20.539970 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zr7w4_114ffe72-9ce9-4eed-baea-d3667e969148/manager/0.log" Oct 30 03:11:21.250545 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:21.250503 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-wptrw_1a3c5537-f83e-412e-a917-27fd46cfe258/manager/0.log" Oct 30 03:11:21.515383 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:21.515303 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9fjjh_712fc2f5-d537-4e96-9801-963522329de7/server/0.log" Oct 30 03:11:21.651639 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:21.651599 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kpkmz_04727d53-1a8f-4f08-9930-acfb77f05eb4/manager/0.log" Oct 30 03:11:23.130965 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:23.130929 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-25r9k_7994cb01-5d42-4871-ae17-d9622123f971/manager/0.log" Oct 30 03:11:23.204316 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:23.204285 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-25r9k_7994cb01-5d42-4871-ae17-d9622123f971/manager/0.log" Oct 30 03:11:23.259581 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:23.259550 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zr7w4_114ffe72-9ce9-4eed-baea-d3667e969148/manager/0.log" Oct 30 03:11:23.343222 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:23.343181 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zr7w4_114ffe72-9ce9-4eed-baea-d3667e969148/manager/0.log" Oct 30 03:11:24.216308 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:24.216263 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-wptrw_1a3c5537-f83e-412e-a917-27fd46cfe258/manager/0.log" Oct 30 03:11:24.340186 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:24.340151 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-wptrw_1a3c5537-f83e-412e-a917-27fd46cfe258/manager/0.log" Oct 30 03:11:24.413346 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:24.413312 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-z4b6g_db689d73-bcb1-41fb-8507-84109c764878/manager/0.log" Oct 30 03:11:24.534145 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:24.534051 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9fjjh_712fc2f5-d537-4e96-9801-963522329de7/server/0.log" Oct 30 03:11:24.677881 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:24.677839 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kpkmz_04727d53-1a8f-4f08-9930-acfb77f05eb4/manager/0.log" Oct 30 03:11:24.706571 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:24.706536 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-s228v_6d3c1f40-053d-4b00-83a2-a6308a90a0b8/manager/0.log" Oct 30 03:11:24.777914 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:24.777880 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9fjjh_712fc2f5-d537-4e96-9801-963522329de7/server/0.log" Oct 30 03:11:24.808732 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:24.808653 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-p2m8w_db342fa8-f998-43b9-8e0a-5da7612cdee2/limitador/0.log" Oct 30 03:11:24.912003 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:24.911973 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kpkmz_04727d53-1a8f-4f08-9930-acfb77f05eb4/manager/0.log" Oct 30 03:11:25.216024 ip-10-0-130-0 kubenswrapper[5998]: E1030 03:11:25.215985 5998 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vccw5" containerName="extractor" Oct 30 03:11:26.246937 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.246896 5998 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72"] Oct 30 03:11:26.247489 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.247431 5998 cpu_manager.go:414] "containerMap: removing container" podUID="cfdd2ee5-7258-4144-a440-6b085d1c2a4e" containerName="revoke-keys" Oct 30 03:11:26.247489 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.247450 5998 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfdd2ee5-7258-4144-a440-6b085d1c2a4e" containerName="revoke-keys" Oct 30 03:11:26.247593 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.247543 5998 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfdd2ee5-7258-4144-a440-6b085d1c2a4e" containerName="revoke-keys" Oct 30 03:11:26.250598 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.250580 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:26.271703 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.271676 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72"] Oct 30 03:11:26.280311 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.280263 5998 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wd7bs\"/\"default-dockercfg-mzxnn\"" Oct 30 03:11:26.280440 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.280327 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wd7bs\"/\"openshift-service-ca.crt\"" Oct 30 03:11:26.280440 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.280399 5998 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wd7bs\"/\"kube-root-ca.crt\"" Oct 30 03:11:26.316232 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.316199 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frbsk\" (UniqueName: \"kubernetes.io/projected/3410eb4b-a40f-4aec-a804-43c36c11ac9b-kube-api-access-frbsk\") pod \"perf-node-gather-daemonset-7fd72\" (UID: \"3410eb4b-a40f-4aec-a804-43c36c11ac9b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:26.316401 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.316252 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3410eb4b-a40f-4aec-a804-43c36c11ac9b-lib-modules\") pod \"perf-node-gather-daemonset-7fd72\" (UID: \"3410eb4b-a40f-4aec-a804-43c36c11ac9b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:26.316401 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.316345 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3410eb4b-a40f-4aec-a804-43c36c11ac9b-sys\") pod \"perf-node-gather-daemonset-7fd72\" (UID: \"3410eb4b-a40f-4aec-a804-43c36c11ac9b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:26.316496 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.316404 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3410eb4b-a40f-4aec-a804-43c36c11ac9b-podres\") pod \"perf-node-gather-daemonset-7fd72\" (UID: \"3410eb4b-a40f-4aec-a804-43c36c11ac9b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:26.316496 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.316481 5998 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3410eb4b-a40f-4aec-a804-43c36c11ac9b-proc\") pod \"perf-node-gather-daemonset-7fd72\" (UID: \"3410eb4b-a40f-4aec-a804-43c36c11ac9b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:26.417424 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.417378 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frbsk\" (UniqueName: \"kubernetes.io/projected/3410eb4b-a40f-4aec-a804-43c36c11ac9b-kube-api-access-frbsk\") pod \"perf-node-gather-daemonset-7fd72\" (UID: \"3410eb4b-a40f-4aec-a804-43c36c11ac9b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:26.417424 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.417430 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3410eb4b-a40f-4aec-a804-43c36c11ac9b-lib-modules\") pod \"perf-node-gather-daemonset-7fd72\" (UID: \"3410eb4b-a40f-4aec-a804-43c36c11ac9b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:26.417689 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.417458 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3410eb4b-a40f-4aec-a804-43c36c11ac9b-sys\") pod \"perf-node-gather-daemonset-7fd72\" (UID: \"3410eb4b-a40f-4aec-a804-43c36c11ac9b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:26.417689 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.417496 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3410eb4b-a40f-4aec-a804-43c36c11ac9b-podres\") pod \"perf-node-gather-daemonset-7fd72\" (UID: \"3410eb4b-a40f-4aec-a804-43c36c11ac9b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:26.417689 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.417529 5998 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3410eb4b-a40f-4aec-a804-43c36c11ac9b-proc\") pod \"perf-node-gather-daemonset-7fd72\" (UID: \"3410eb4b-a40f-4aec-a804-43c36c11ac9b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:26.417689 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.417569 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3410eb4b-a40f-4aec-a804-43c36c11ac9b-lib-modules\") pod \"perf-node-gather-daemonset-7fd72\" (UID: \"3410eb4b-a40f-4aec-a804-43c36c11ac9b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:26.417689 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.417595 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3410eb4b-a40f-4aec-a804-43c36c11ac9b-sys\") pod \"perf-node-gather-daemonset-7fd72\" (UID: \"3410eb4b-a40f-4aec-a804-43c36c11ac9b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:26.417689 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.417612 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3410eb4b-a40f-4aec-a804-43c36c11ac9b-proc\") pod \"perf-node-gather-daemonset-7fd72\" (UID: \"3410eb4b-a40f-4aec-a804-43c36c11ac9b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:26.417689 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.417627 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3410eb4b-a40f-4aec-a804-43c36c11ac9b-podres\") pod \"perf-node-gather-daemonset-7fd72\" (UID: \"3410eb4b-a40f-4aec-a804-43c36c11ac9b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:26.440308 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.440282 5998 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frbsk\" (UniqueName: \"kubernetes.io/projected/3410eb4b-a40f-4aec-a804-43c36c11ac9b-kube-api-access-frbsk\") pod \"perf-node-gather-daemonset-7fd72\" (UID: \"3410eb4b-a40f-4aec-a804-43c36c11ac9b\") " pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:26.563157 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.563054 5998 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:26.687779 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.687749 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-25r9k_7994cb01-5d42-4871-ae17-d9622123f971/manager/0.log" Oct 30 03:11:26.797426 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.797393 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zr7w4_114ffe72-9ce9-4eed-baea-d3667e969148/manager/0.log" Oct 30 03:11:26.864769 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:26.864677 5998 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72"] Oct 30 03:11:26.868410 ip-10-0-130-0 kubenswrapper[5998]: W1030 03:11:26.868375 5998 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3410eb4b_a40f_4aec_a804_43c36c11ac9b.slice/crio-5076ead212624d68f08f08863e73b32cd39c5eaadf8c728f14288b9c0c5179e1 WatchSource:0}: Error finding container 5076ead212624d68f08f08863e73b32cd39c5eaadf8c728f14288b9c0c5179e1: Status 404 returned error can't find the container with id 5076ead212624d68f08f08863e73b32cd39c5eaadf8c728f14288b9c0c5179e1 Oct 30 03:11:27.574454 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:27.574409 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-wptrw_1a3c5537-f83e-412e-a917-27fd46cfe258/manager/0.log" Oct 30 03:11:27.790891 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:27.790850 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" event={"ID":"3410eb4b-a40f-4aec-a804-43c36c11ac9b","Type":"ContainerStarted","Data":"6fc66064ce095a9e7c5a06377cf12f24678543362ae1dd8a8280e878f0bc0ffd"} Oct 30 03:11:27.791083 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:27.790951 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:27.791083 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:27.790974 5998 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" event={"ID":"3410eb4b-a40f-4aec-a804-43c36c11ac9b","Type":"ContainerStarted","Data":"5076ead212624d68f08f08863e73b32cd39c5eaadf8c728f14288b9c0c5179e1"} Oct 30 03:11:27.856043 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:27.855940 5998 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" podStartSLOduration=1.855923877 podStartE2EDuration="1.855923877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:27.85588762 +0000 UTC m=+6117.434678702" watchObservedRunningTime="2026-10-30 03:11:27.855923877 +0000 UTC m=+6117.434714958" Oct 30 03:11:28.092236 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:28.092201 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9fjjh_712fc2f5-d537-4e96-9801-963522329de7/server/0.log" Oct 30 03:11:28.097392 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:28.097364 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-868b7dc86d-hzrmz_3657b83a-1cdf-44de-9a9d-2800cfae5524/metrics-server/0.log" Oct 30 03:11:28.246876 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:28.246839 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kpkmz_04727d53-1a8f-4f08-9930-acfb77f05eb4/manager/0.log" Oct 30 03:11:28.893761 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:28.893727 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bd6fc9ff9-w5zwp_746a1452-89c9-4b38-9767-318905eaf8fe/kube-auth-proxy/0.log" Oct 30 03:11:28.962946 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:28.962912 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-28ssd_d8da8b22-5586-419c-9378-03d3d2d2c47f/istio-proxy/0.log" Oct 30 03:11:29.054829 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:29.054794 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nd754_b4c33058-34c5-49f3-b803-984c54ff5dad/node-exporter/0.log" Oct 30 03:11:29.141119 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:29.141065 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-b95w4_0e843d95-ed87-4ea3-9ce8-d3f0103d9a80/istio-proxy/0.log" Oct 30 03:11:29.171540 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:29.171519 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nd754_b4c33058-34c5-49f3-b803-984c54ff5dad/kube-rbac-proxy/0.log" Oct 30 03:11:29.288987 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:29.288959 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nd754_b4c33058-34c5-49f3-b803-984c54ff5dad/init-textfile/0.log" Oct 30 03:11:29.376875 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:29.376841 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-ppvtr_961683fb-469d-4d4a-9463-269eb264251a/istio-proxy/0.log" Oct 30 03:11:29.415377 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:29.415345 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8dtpq_43e8ec8c-149e-4257-a891-7b117a79c319/kube-rbac-proxy-main/0.log" Oct 30 03:11:29.534120 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:29.534028 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8dtpq_43e8ec8c-149e-4257-a891-7b117a79c319/kube-rbac-proxy-self/0.log" Oct 30 03:11:29.555271 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:29.555245 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q_b0a8f11c-0b47-448d-9c4a-d98899aabf93/istio-proxy/0.log" Oct 30 03:11:29.650848 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:29.650818 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8dtpq_43e8ec8c-149e-4257-a891-7b117a79c319/openshift-state-metrics/0.log" Oct 30 03:11:29.784079 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:29.784050 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0b5239b-d668-4a7a-b836-9bebc978f33c/prometheus/0.log" Oct 30 03:11:29.793217 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:29.793153 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-25r9k_7994cb01-5d42-4871-ae17-d9622123f971/manager/0.log" Oct 30 03:11:29.900928 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:29.900894 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0b5239b-d668-4a7a-b836-9bebc978f33c/config-reloader/0.log" Oct 30 03:11:29.913469 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:29.913441 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-zr7w4_114ffe72-9ce9-4eed-baea-d3667e969148/manager/0.log" Oct 30 03:11:30.020421 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:30.020388 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0b5239b-d668-4a7a-b836-9bebc978f33c/thanos-sidecar/0.log" Oct 30 03:11:30.146015 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:30.145925 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0b5239b-d668-4a7a-b836-9bebc978f33c/kube-rbac-proxy-web/0.log" Oct 30 03:11:30.264200 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:30.264166 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0b5239b-d668-4a7a-b836-9bebc978f33c/kube-rbac-proxy/0.log" Oct 30 03:11:30.382895 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:30.382864 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0b5239b-d668-4a7a-b836-9bebc978f33c/kube-rbac-proxy-thanos/0.log" Oct 30 03:11:30.499721 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:30.499692 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c0b5239b-d668-4a7a-b836-9bebc978f33c/init-config-reloader/0.log" Oct 30 03:11:31.166372 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:31.166332 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-wptrw_1a3c5537-f83e-412e-a917-27fd46cfe258/manager/0.log" Oct 30 03:11:31.320905 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:31.320872 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-767946cb5-q64ns_4fa1c18e-5933-4e09-85f8-28033e6cc4df/thanos-query/0.log" Oct 30 03:11:31.438253 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:31.438171 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-767946cb5-q64ns_4fa1c18e-5933-4e09-85f8-28033e6cc4df/kube-rbac-proxy-web/0.log" Oct 30 03:11:31.555688 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:31.555657 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-767946cb5-q64ns_4fa1c18e-5933-4e09-85f8-28033e6cc4df/kube-rbac-proxy/0.log" Oct 30 03:11:31.622986 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:31.622957 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-9fjjh_712fc2f5-d537-4e96-9801-963522329de7/server/0.log" Oct 30 03:11:31.675509 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:31.675475 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-767946cb5-q64ns_4fa1c18e-5933-4e09-85f8-28033e6cc4df/prom-label-proxy/0.log" Oct 30 03:11:31.756987 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:31.756899 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kpkmz_04727d53-1a8f-4f08-9930-acfb77f05eb4/manager/0.log" Oct 30 03:11:31.792696 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:31.792669 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-767946cb5-q64ns_4fa1c18e-5933-4e09-85f8-28033e6cc4df/kube-rbac-proxy-rules/0.log" Oct 30 03:11:31.911032 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:31.911002 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-767946cb5-q64ns_4fa1c18e-5933-4e09-85f8-28033e6cc4df/kube-rbac-proxy-metrics/0.log" Oct 30 03:11:33.807980 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:33.807946 5998 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wd7bs/perf-node-gather-daemonset-7fd72" Oct 30 03:11:34.127386 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:34.127299 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-z4b6g_db689d73-bcb1-41fb-8507-84109c764878/manager/0.log" Oct 30 03:11:34.298665 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:34.298630 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-s228v_6d3c1f40-053d-4b00-83a2-a6308a90a0b8/manager/0.log" Oct 30 03:11:34.404133 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:34.404036 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-p2m8w_db342fa8-f998-43b9-8e0a-5da7612cdee2/limitador/0.log" Oct 30 03:11:35.511237 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:35.511201 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bd6fc9ff9-w5zwp_746a1452-89c9-4b38-9767-318905eaf8fe/kube-auth-proxy/0.log" Oct 30 03:11:35.585927 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:35.585885 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-28ssd_d8da8b22-5586-419c-9378-03d3d2d2c47f/istio-proxy/0.log" Oct 30 03:11:35.773966 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:35.773890 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-b95w4_0e843d95-ed87-4ea3-9ce8-d3f0103d9a80/istio-proxy/0.log" Oct 30 03:11:36.014017 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:36.013988 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-ppvtr_961683fb-469d-4d4a-9463-269eb264251a/istio-proxy/0.log" Oct 30 03:11:36.201303 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:36.201275 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q_b0a8f11c-0b47-448d-9c4a-d98899aabf93/istio-proxy/0.log" Oct 30 03:11:40.384606 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:40.384542 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6vkc5_da638295-de7f-462a-9a14-2b8105e636b6/dns/0.log" Oct 30 03:11:40.504893 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:40.504850 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6vkc5_da638295-de7f-462a-9a14-2b8105e636b6/kube-rbac-proxy/0.log" Oct 30 03:11:41.206193 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:41.206167 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8wd8v_174be2db-9c13-45fe-b91f-c3302ecb57e2/dns-node-resolver/0.log" Oct 30 03:11:41.443058 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:41.443005 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-z4b6g_db689d73-bcb1-41fb-8507-84109c764878/manager/0.log" Oct 30 03:11:41.604726 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:41.604634 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-s228v_6d3c1f40-053d-4b00-83a2-a6308a90a0b8/manager/0.log" Oct 30 03:11:41.754577 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:41.754501 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-p2m8w_db342fa8-f998-43b9-8e0a-5da7612cdee2/limitador/0.log" Oct 30 03:11:42.871378 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:42.871344 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bd6fc9ff9-w5zwp_746a1452-89c9-4b38-9767-318905eaf8fe/kube-auth-proxy/0.log" Oct 30 03:11:42.935607 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:42.935573 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-28ssd_d8da8b22-5586-419c-9378-03d3d2d2c47f/istio-proxy/0.log" Oct 30 03:11:43.019821 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:43.019788 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lq7lj_4f486e9e-aa32-4a9a-877b-eca8dbba7091/node-ca/0.log" Oct 30 03:11:43.121811 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:43.121727 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-b95w4_0e843d95-ed87-4ea3-9ce8-d3f0103d9a80/istio-proxy/0.log" Oct 30 03:11:43.374144 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:43.374052 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-ppvtr_961683fb-469d-4d4a-9463-269eb264251a/istio-proxy/0.log" Oct 30 03:11:43.550843 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:43.550809 5998 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cf4665f7f-8ml9q_b0a8f11c-0b47-448d-9c4a-d98899aabf93/istio-proxy/0.log" Oct 30 03:11:44.978272 ip-10-0-130-0 kubenswrapper[5998]: I1030 03:11:44.978238 5998 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-44ftt_a0d9a754-4e82-4dd5-b423-50624589bdda/konnectivity-agent/0.log" limit=1024